Nov 03 11:50:07.176407 ip-10-0-131-136 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 11:50:07.582337 ip-10-0-131-136 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:07.582337 ip-10-0-131-136 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 11:50:07.582337 ip-10-0-131-136 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:07.582337 ip-10-0-131-136 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:07.588674 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588585    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.588674 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588669    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.588674 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588674    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.588674 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588678    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.588674 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588682    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588686    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588691    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588694    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588705    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588716    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588720    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588724    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588728    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588732    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588736    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588740    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588744    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588747    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588751    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588756    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588760    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588764    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588768    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.588968 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588771    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588775    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588779    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588783    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588787    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588791    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588796    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588800    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588804    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588809    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588813    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588817    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588822    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588826    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588831    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588835    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588841    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588845    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588850    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588854    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.589746 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588859    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588864    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588868    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588872    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588879    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588884    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588890    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588895    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588899    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588904    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588909    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588913    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588917    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588921    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588926    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588930    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588936    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588941    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588945    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.590604 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588949    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588954    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588958    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588962    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588968    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588975    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588980    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588984    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588989    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588994    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.588999    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589004    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589008    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589013    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589017    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589021    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589025    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589030    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589034    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.591416 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589038    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589042    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589047    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589051    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589055    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589059    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589064    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589068    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589072    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589076    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589081    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589085    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589089    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589108    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589112    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589117    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589121    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589125    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589129    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589133    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.592365 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589138    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589141    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589146    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589151    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.589158    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590412    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590422    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590427    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590431    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590435    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590439    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590444    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590448    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590453    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590457    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590464    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590472    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590476    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590481    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.592850 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590486    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590490    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590494    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590498    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590503    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590507    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590511    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590516    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590520    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590524    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590528    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590532    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590537    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590541    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590545    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590549    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590554    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590558    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590563    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590567    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.593506 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590571    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590577    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590582    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590587    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590591    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590595    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590599    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590603    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590607    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590610    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590614    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590621    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590625    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590629    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590634    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590639    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590644    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590649    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590653    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.594377 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590657    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590662    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590666    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590670    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590675    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590678    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590683    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590687    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590691    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590695    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590700    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590704    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590709    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590714    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590719    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590723    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590728    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590732    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590736    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.595285 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590740    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590744    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590748    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590753    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590756    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590761    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590766    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590770    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590774    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590779    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590783    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590790    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590794    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590798    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590802    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590807    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590811    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590815    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590819    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.595826 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590823    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590827    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590832    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590836    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590840    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590844    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590849    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590854    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590858    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590863    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590867    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590873    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590877    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590882    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.590885    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591719    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591728    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591734    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591739    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591743    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.596497 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591748    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591752    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591758    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591762    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591766    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591771    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591775    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591778    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591783    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591787    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591791    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591795    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591799    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591804    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591808    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591812    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591816    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591820    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591823    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591829    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.597012 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591833    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591838    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591842    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591846    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591850    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591853    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591857    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591861    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591865    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591869    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591873    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591877    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591881    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591885    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591889    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591893    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591898    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591901    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591905    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.597492 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591909    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591913    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591917    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591921    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591925    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591929    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591933    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591936    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591940    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591945    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591949    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591953    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591957    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591962    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591967    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591971    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591975    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591980    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591984    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.598023 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591988    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591993    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.591997    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592001    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592005    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592012    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592017    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592021    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592026    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592030    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592035    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592039    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592043    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592051    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592057    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592061    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592066    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592070    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592075    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.598491 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592079    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592084    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592088    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592109    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592114    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592118    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592122    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592126    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592132    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592137    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592141    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592145    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592149    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592154    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592158    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592162    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592167    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592171    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592176    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592180    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.598966 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592184    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592188    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592193    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.592197    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593173    2581 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593197    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593208    2581 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593215    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593222    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593227    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593234    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593240    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593245    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593250    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593256    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593261    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593266    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593271    2581 flags.go:64] FLAG: --cgroup-root=""
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593275    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593280    2581 flags.go:64] FLAG: --client-ca-file=""
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593285    2581 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593291    2581 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593296    2581 flags.go:64] FLAG: --cluster-domain=""
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593301    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 11:50:07.599458 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593306    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593311    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593317    2581 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593323    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593327    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593333    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593337    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593342    2581 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593350    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593355    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593360    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593365    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593372    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593377    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593382    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593387    2581 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593391    2581 flags.go:64] FLAG: --enable-server="true"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593396    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593406    2581 flags.go:64] FLAG: --event-burst="100"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593411    2581 flags.go:64] FLAG: --event-qps="50"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593415    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593421    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593425    2581 flags.go:64] FLAG: --eviction-hard=""
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593432    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 11:50:07.600002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593437    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593442    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593447    2581 flags.go:64] FLAG: --eviction-soft=""
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593452    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593457    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593461    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593467    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593472    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593477    2581 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593482    2581 flags.go:64] FLAG: --feature-gates=""
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593488    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593493    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593498    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593503    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593508    2581 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593513    2581 flags.go:64] FLAG: --help="false"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593519    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593524    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593529    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593534    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593539    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593545    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593550    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 11:50:07.600642 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593554    2581 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593559    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593564    2581 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593569    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593574    2581 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593578    2581 flags.go:64] FLAG: --kube-reserved=""
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593583    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593588    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593593    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593597    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593602    2581 flags.go:64] FLAG: --lock-file=""
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593606    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593611    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593618    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593628    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593633    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593643    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593648    2581 flags.go:64] FLAG: --logging-format="text"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593653    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593658    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593662    2581 flags.go:64] FLAG: --manifest-url=""
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593667    2581 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593674    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593679    2581 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593685    2581 flags.go:64] FLAG: --max-pods="110"
Nov 03 11:50:07.601234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593692    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593697    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593702    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593706    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593711    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593717    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593721    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593732    2581 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593738    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593742    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593747    2581 flags.go:64] FLAG: --pod-cidr=""
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593752    2581 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593756    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593761    2581 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593766    2581 flags.go:64] FLAG: --port="10250"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593771    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593776    2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-02c4211575cb30411"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593781    2581 flags.go:64] FLAG: --qos-reserved=""
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593786    2581 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593791    2581 flags.go:64] FLAG: --register-node="true"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593800    2581 flags.go:64] FLAG: --register-with-taints=""
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593806    2581 flags.go:64] FLAG: --registry-burst="10"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593811    2581 flags.go:64] FLAG: --registry-qps="5"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593815    2581 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593820    2581 flags.go:64] FLAG: --reserved-memory=""
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593826    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 11:50:07.601821 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593832    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593837    2581 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593841    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593846    2581 flags.go:64] FLAG: --runonce="false"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593851    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593856    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593862    2581 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593867    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593872    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593877    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593882    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593887    2581 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593892    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593896    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593901    2581 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593906    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593911    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593916    2581 flags.go:64] FLAG: --system-cgroups=""
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593920    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593936    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593940    2581 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593945    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593950    2581 flags.go:64] FLAG: --tls-min-version=""
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593955    2581 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593960    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 11:50:07.602889 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593965    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593971    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593976    2581 flags.go:64] FLAG: --v="2"
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593989    2581 flags.go:64] FLAG: --version="false"
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.593996    2581 flags.go:64] FLAG: --vmodule=""
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.594003    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.594009    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594239    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594246    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594250    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594255    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594260    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594264    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594271    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594275    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594279    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594283    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594288    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594292    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594296    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594301    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.603562 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594305    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594310    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594315    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594320    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594324    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594328    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594332    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594336    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594341    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594345    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594349    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594353    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594359    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594363    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594367    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594372    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594375    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594380    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594386    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594390    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.604063 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594394    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594398    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594402    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594406    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594412    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594417    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594421    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594425    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594429    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594434    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594438    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594442    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594446    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594450    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594454    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594459    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594463    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594470    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594475    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.604563 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594482    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594487    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594493    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594498    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594503    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594509    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594513    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594518    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594522    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594526    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594531    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594534    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594540    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594544    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594549    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594553    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594557    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594562    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594567    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.605049 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594573    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594579    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594585    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594590    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594595    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594600    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594604    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594608    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594613    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594617    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594621    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594626    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594630    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594634    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594638    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594643    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594647    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594651    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594656    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594661    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.605514 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594665    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594669    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594674    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594678    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594684    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594689    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594694    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594699    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594703    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594708    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594714    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594718    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594723    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594727    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.594771    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594829    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.605971 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594836    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594841    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594845    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594851    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594855    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594860    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594864    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594869    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594873    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594877    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594881    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594887    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594891    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594895    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594902    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594906    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594910    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594914    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594919    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594923    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.606374 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594927    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594932    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594936    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594941    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594945    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594949    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594953    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594959    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594963    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594968    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594972    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594977    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594981    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594986    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594990    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594994    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.594998    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595003    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595007    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.606844 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595012    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595016    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595020    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595024    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595028    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595033    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595037    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595043    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595047    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595054    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595059    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595063    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595068    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595075    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595079    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595083    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595088    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595111    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595116    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.607308 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595120    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595127    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595131    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595136    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595140    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595144    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595148    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595152    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595156    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595160    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595164    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595171    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595177    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595181    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595186    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595191    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595195    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595199    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595203    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.607754 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595207    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595214    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595218    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595222    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595226    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595231    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595235    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595239    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595243    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595248    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595252    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595257    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595262    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595266    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595272    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595277    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595281    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595285    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595290    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595293    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.608280 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595297    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595302    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595306    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595310    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595315    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595319    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595326    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.595330    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.603942    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.603959    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.604695    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.604708    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604765    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604771    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604775    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604778    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604781    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604784    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604787    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.608747 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604790    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604792    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604795    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604798    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604801    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604803    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604806    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604808    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604811    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604814    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604817    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604819    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604822    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604824    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604827    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604829    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604832    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604834    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604837    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.609213 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604839    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604844    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604848    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604852    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604857    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604860    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604863    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604866    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604869    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604872    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604874    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604877    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604880    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604882    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604885    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604888    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604890    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604893    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604895    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604898    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.609651 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604907    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604909    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604912    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604915    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604917    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604920    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604922    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604925    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604927    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604929    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604932    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604934    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604937    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604939    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604942    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604946    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604948    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604951    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604954    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.610205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604956    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604959    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604961    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604964    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604967    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604969    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604972    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604975    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604978    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604980    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604983    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604985    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604988    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.604990    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605000    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605003    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605005    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605008    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605011    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605013    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.610644 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605016    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605019    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605021    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605023    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605026    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605029    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605031    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605034    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605037    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605040    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605043    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605046    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605048    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605051    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605054    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605057    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605059    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605062    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605065    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.611108 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605067    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605070    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.605113    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605255    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605261    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605266    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605270    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605273    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605282    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605285    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605287    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605290    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605293    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605295    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605298    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.611549 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605300    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605303    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605305    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605308    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605311    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605313    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605316    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605319    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605322    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605324    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605327    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605329    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605332    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605335    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605337    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605340    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605342    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605345    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605347    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605350    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.611917 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605352    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605355    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605357    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605360    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605363    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605365    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605369    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605371    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605374    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605376    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605379    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605382    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605384    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605387    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605389    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605392    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605394    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605397    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605399    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605402    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.612460 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605404    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605407    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605409    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605412    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605414    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605417    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605419    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605422    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605424    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605427    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605429    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605432    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605434    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605437    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605439    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605442    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605444    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605447    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605449    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605452    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.612940 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605455    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605457    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605460    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605463    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605465    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605468    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605470    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605473    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605476    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605478    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605481    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605483    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605486    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605488    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605492    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605494    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605497    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605499    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605502    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.613522 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605505    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605507    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605510    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605512    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605515    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605518    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605520    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605523    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605525    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605528    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605531    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605533    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605536    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605539    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605541    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.605565    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:07.613972 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605595    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605600    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605603    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605606    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605609    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605612    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605615    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605618    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605621    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605624    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605626    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605628    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605631    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605633    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605636    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605639    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605641    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605644    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605646    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605649    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:07.614366 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605675    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605679    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605682    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605686    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605690    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605695    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605698    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605700    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605703    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605706    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605709    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605711    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605714    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605716    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605719    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605721    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605724    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605726    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605729    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:07.614834 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605731    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605734    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605737    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605740    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605742    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605744    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605747    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605749    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605752    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605755    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605757    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605760    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605762    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605765    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605767    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605769    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605772    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605775    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605777    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605780    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:07.615304 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605782    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605785    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605787    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605790    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605793    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605796    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605798    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605801    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605804    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605806    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605809    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605811    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605814    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605816    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605819    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605821    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605823    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605826    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605829    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605831    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:07.615931 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605834    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605837    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605839    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605842    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605844    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605847    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605849    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605852    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605854    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605857    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605859    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605862    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605864    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605867    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605869    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605872    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605874    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605879    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605882    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605885    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:07.616489 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605888    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605890    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605894    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605897    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605899    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605902    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:07.605904    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.605960    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.609762    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.610594    2581 server.go:1014] "Starting client certificate rotation"
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.610686    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.610725    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.612480    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:07.616985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.615736    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:07.626748 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.626732    2581 log.go:25] "Validated CRI v1 runtime API"
Nov 03 11:50:07.630666 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.630653    2581 log.go:25] "Validated CRI v1 image API"
Nov 03 11:50:07.631497 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.631474    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 11:50:07.636523 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.636501    2581 fs.go:135] Filesystem UUIDs: map[43b3e96e-0519-47e4-803f-06f2b5b72e14:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cffe2e66-271f-42fb-90a6-bb71261a2220:/dev/nvme0n1p3]
Nov 03 11:50:07.636617 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.636522    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 11:50:07.642719 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.642607    2581 manager.go:217] Machine: {Timestamp:2026-11-03 11:50:07.641057954 +0000 UTC m=+0.344153736 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083598 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22541a0be8d026063352a07a9ddc80 SystemUUID:ec22541a-0be8-d026-0633-52a07a9ddc80 BootID:f664fcb4-09b1-4ab4-a892-26d870e046ba Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f9:a8:c1:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f9:a8:c1:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:44:c7:1c:d4:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 11:50:07.643380 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.643370    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 11:50:07.643460 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.643444    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 11:50:07.646352 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.646331    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 11:50:07.646548 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.646361    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-136.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 11:50:07.647142 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.647132    2581 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 11:50:07.647171 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.647145    2581 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 11:50:07.647207 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.647170    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:07.647207 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.647186    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:07.648159 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.648149    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 11:50:07.652206 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.652194    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 11:50:07.653375 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.653285    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 11:50:07.654302 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.654287    2581 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 11:50:07.654386 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.654311    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 11:50:07.654386 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.654329    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 11:50:07.654386 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.654343    2581 kubelet.go:405] "Adding apiserver pod source"
Nov 03 11:50:07.654386 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.654354    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 11:50:07.654689 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.654675    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:07.657009 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.656992    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 11:50:07.663321 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.663307    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:07.664184 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664158    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 11:50:07.664269 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664210    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 11:50:07.664404 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664391    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 11:50:07.664467 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664410    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 11:50:07.664467 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664425    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 11:50:07.664467 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664434    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 11:50:07.664467 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664443    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 11:50:07.664467 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.664452    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 11:50:07.665158 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665146    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 11:50:07.665215 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665161    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 11:50:07.665215 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665172    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 11:50:07.665215 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665181    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 11:50:07.665215 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665194    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 11:50:07.665215 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665208    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 11:50:07.665881 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665870    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 11:50:07.665935 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.665886    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 11:50:07.671036 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.671018    2581 server.go:1263] "Started kubelet"
Nov 03 11:50:07.671127 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.671088    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 11:50:07.671179 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.671123    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 11:50:07.671237 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.671197    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 11:50:07.671706 ip-10-0-131-136 systemd[1]: Started Kubernetes Kubelet.
Nov 03 11:50:07.672319 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.672302    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 11:50:07.672985 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.672974    2581 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 11:50:07.678771 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.678744    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 11:50:07.678854 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.678810    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:07.679352 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.679332    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 11:50:07.679441 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.679425    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.us-west-2.compute.internal\" not found"
Nov 03 11:50:07.679982 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.679966    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 11:50:07.679982 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.679970    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 11:50:07.680124 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.679992    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 11:50:07.680124 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680081    2581 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 11:50:07.680124 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680091    2581 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 11:50:07.680851 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680836    2581 factory.go:153] Registering CRI-O factory
Nov 03 11:50:07.680851 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680852    2581 factory.go:223] Registration of the crio container factory successfully
Nov 03 11:50:07.680997 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680889    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 11:50:07.680997 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680898    2581 factory.go:55] Registering systemd factory
Nov 03 11:50:07.680997 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680904    2581 factory.go:223] Registration of the systemd container factory successfully
Nov 03 11:50:07.680997 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680915    2581 factory.go:103] Registering Raw factory
Nov 03 11:50:07.680997 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.680923    2581 manager.go:1199] Started watching for new ooms in manager
Nov 03 11:50:07.681912 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.681894    2581 manager.go:319] Starting recovery of all containers
Nov 03 11:50:07.692322 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.692305    2581 manager.go:324] Recovery completed
Nov 03 11:50:07.696557 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.696542    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:07.698329 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.698312    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:07.698416 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.698347    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:07.698416 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.698363    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:07.698921 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.698906    2581 cpu_manager.go:226] "Starting" policy="none"
Nov 03 11:50:07.698921 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.698917    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 11:50:07.698921 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.698920    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:07.699084 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.698934    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 11:50:07.699593 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.699580    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:07.699659 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.699609    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:07.699659 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.699624    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:07.701037 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.701025    2581 policy_none.go:50] "Start"
Nov 03 11:50:07.701080 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.701041    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 11:50:07.701080 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.701051    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 11:50:07.702963 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.702952    2581 policy_none.go:44] "Start"
Nov 03 11:50:07.728732 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.728707    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 11:50:07.731090 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.731078    2581 manager.go:341] "Starting Device Plugin manager"
Nov 03 11:50:07.731174 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.731120    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 11:50:07.731174 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.731132    2581 server.go:87] "Starting device plugin registration server"
Nov 03 11:50:07.731375 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.731362    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 11:50:07.731416 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.731376    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 11:50:07.731456 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.731443    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 11:50:07.731535 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.731517    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 11:50:07.731535 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.731531    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 11:50:07.732178 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.732145    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 11:50:07.732282 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.732193    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-136.us-west-2.compute.internal\" not found"
Nov 03 11:50:07.810435 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.810404    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 11:50:07.812054 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.810393    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.us-west-2.compute.internal.18e486a7fd8008c5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:07.670986949 +0000 UTC m=+0.374082730,LastTimestamp:2026-11-03 11:50:07.670986949 +0000 UTC m=+0.374082730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.us-west-2.compute.internal,}"
Nov 03 11:50:07.814686 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.814655    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 11:50:07.815871 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.815856    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 11:50:07.815928 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.815874    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 11:50:07.815928 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.815893    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 11:50:07.816008 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.815930    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 11:50:07.831468 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.831453    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:07.832172 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.832157    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:07.832259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.832183    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:07.832259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.832195    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:07.832259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.832224    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.855417 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.855335    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 11:50:07.873303 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.873283    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 11:50:07.873393 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.873328    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kwm6p"
Nov 03 11:50:07.916952 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.916788    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"]
Nov 03 11:50:07.917059 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.917003    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:07.918482 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.918464    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:07.918561 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.918499    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:07.918561 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.918514    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:07.920570 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.920558    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:07.920719 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.920694    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.920758 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.920737    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:07.921343 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.921322    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:07.921343 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.921334    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:07.921456 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.921355    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:07.921456 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.921356    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:07.921456 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.921371    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:07.921456 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.921376    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:07.923292 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.923277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.923349 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.923307    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:07.924416 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.924402    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:07.924508 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.924422    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:07.924508 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.924434    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:07.946160 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.946144    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.us-west-2.compute.internal\" not found" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.948753 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:07.948736    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.us-west-2.compute.internal\" not found" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.957774 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.957753    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:07.959722 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.959708    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:07.981187 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.981166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.981262 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.981192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:07.981262 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:07.981230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2905ce8bf73baad4434061659facba8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"f2905ce8bf73baad4434061659facba8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.050707 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.050675    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:08.053666 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.053642    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:08.055216 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.055196    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.080541 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.080509    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.081670 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.081650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.081735 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.081680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.081735 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.081701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2905ce8bf73baad4434061659facba8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"f2905ce8bf73baad4434061659facba8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.081804 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.081749    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2905ce8bf73baad4434061659facba8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"f2905ce8bf73baad4434061659facba8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.081804 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.081752    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.081804 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.081764    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.180692 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.180618    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:08.180828 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.180721    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.250453 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.250433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.252553 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.252536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:08.282810 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:08.282792    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:09.792708 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:09.792674    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 11:50:09.793145 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:09.792825    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 11:50:09.793145 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:09.792822    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 11:50:09.793145 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:09.792823    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 11:50:09.793145 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:09.792853    2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9e0eeeddb0cc409da088ac8135c41d1-80c4a411ee0df651.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-136.us-west-2.compute.internal.18e486a7ff213c37\": read tcp 10.0.131.136:51074->13.184.82.147:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-136.us-west-2.compute.internal.18e486a7ff213c37  default   4847 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:07 +0000 UTC,LastTimestamp:2026-11-03 11:50:07.918481922 +0000 UTC m=+0.621577706,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.us-west-2.compute.internal,}"
Nov 03 11:50:09.837247 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:09.837232    2581 apiserver.go:52] "Watching apiserver"
Nov 03 11:50:09.862466 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:09.862451    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:10.061396 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.061310    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 11:45:08 +0000 UTC" deadline="2028-06-12 12:09:36.846002888 +0000 UTC"
Nov 03 11:50:10.061396 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.061344    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14088h19m26.784661873s"
Nov 03 11:50:10.100836 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.100814    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 11:50:10.101129 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.101112    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lsvz7","openshift-dns/node-resolver-7bdv2","openshift-multus/multus-dqh56","openshift-network-diagnostics/network-check-target-gkt2k","openshift-multus/multus-additional-cni-plugins-j7bdh","openshift-network-operator/iptables-alerter-8pdls","kube-system/konnectivity-agent-df7t5","kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg","openshift-cluster-node-tuning-operator/tuned-742vk","openshift-image-registry/node-ca-dcljv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-cs8vd","openshift-ovn-kubernetes/ovnkube-node-5qjf7"]
Nov 03 11:50:10.103493 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.103475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.103580 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.103557    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:10.105473 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.105457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.107426 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.107405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.109440 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.109424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:10.109536 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.109485    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:10.111437 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.111422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.113493 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.113477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.115542 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.115527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.117676 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.117661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.119842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.119827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.121892 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.121875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.123953 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.123940    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:10.124021 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.123986    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:10.126155 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.126138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.153257 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.153231    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 11:50:10.153332 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.153306    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9rbdn"
Nov 03 11:50:10.157666 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.157644    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 11:50:10.157861 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.157840    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 11:50:10.158101 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.158086    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xf9nj\""
Nov 03 11:50:10.159533 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.159514    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 11:50:10.159620 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.159603    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 11:50:10.159770 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.159711    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g5g7l\""
Nov 03 11:50:10.160108 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.159785    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 11:50:10.160108 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.160043    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 11:50:10.163213 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.163193    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 11:50:10.163311 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.163255    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 11:50:10.163311 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.163262    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jp68j\""
Nov 03 11:50:10.163550 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.163537    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 11:50:10.165334 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.165318    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vn7bf\""
Nov 03 11:50:10.165411 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.165339    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:10.165587 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.165574    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 11:50:10.165629 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.165591    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:10.167076 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.167058    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 11:50:10.167223 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.167211    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 11:50:10.167364 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.167353    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-swh4m\""
Nov 03 11:50:10.169166 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cnibin\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.169246 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.169246 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-hostroot\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.169246 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-multus-certs\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.169393 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smlbr\" (UniqueName: \"kubernetes.io/projected/bfd61052-ba77-4b91-bb1b-9d808791b96b-kube-api-access-smlbr\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.169393 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-systemd-units\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.169393 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-netns\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.169393 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.169393 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdzg\" (UniqueName: \"kubernetes.io/projected/874b36e5-1db3-498f-a51c-e23ccfb5947b-kube-api-access-ccdzg\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.169393 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-sys-fs\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.169608 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.169608 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9t6r\" (UniqueName: \"kubernetes.io/projected/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-kube-api-access-j9t6r\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.169608 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-system-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.169608 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-etc-kubernetes\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.169608 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-conf\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.169608 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-systemd\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.169608 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-script-lib\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-registration-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d378c41e-4f5f-4f08-b051-23694fee59d4-host\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-bin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169653    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-conf-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-run\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169722    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169747    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169760    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6gwtx\""
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-host-slash\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d378c41e-4f5f-4f08-b051-23694fee59d4-serviceca\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-netns\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/77d106d7-59a7-459c-b012-aaa087cb50d9-konnectivity-ca\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.169870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-node-log\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq6q\" (UniqueName: \"kubernetes.io/projected/d378c41e-4f5f-4f08-b051-23694fee59d4-kube-api-access-nnq6q\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-kubernetes\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-log-socket\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovn-node-metrics-cert\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-device-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkqw5\" (UniqueName: \"kubernetes.io/projected/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-kube-api-access-xkqw5\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.169999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-k8s-cni-cncf-io\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-tmp\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-iptables-alerter-script\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-slash\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-env-overrides\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-hosts-file\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.170259 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-daemon-config\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/77d106d7-59a7-459c-b012-aaa087cb50d9-agent-certs\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-sys\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-config\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxwp\" (UniqueName: \"kubernetes.io/projected/b9319607-a6c8-4003-89bd-14da12ba17a8-kube-api-access-hzxwp\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-socket-dir-parent\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-modprobe-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-kubelet\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-var-lib-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-etc-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-ovn\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-tmp-dir\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-kubelet\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.170677 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-lib-modules\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-tuned\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-dbus\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-cnibin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxpx9\" (UniqueName: \"kubernetes.io/projected/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-kube-api-access-zxpx9\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mn4m\" (UniqueName: \"kubernetes.io/projected/a208df1b-dba2-4b61-bd72-d7a4dffcada0-kube-api-access-5mn4m\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-cni-binary-copy\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-bin\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-kubelet-config\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-socket-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktl5\" (UniqueName: \"kubernetes.io/projected/9ceda7ca-9770-42ec-973a-0435ff74dd29-kube-api-access-lktl5\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysconfig\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-var-lib-kubelet\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-systemd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.171136 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-netd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.171563 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-system-cni-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.171563 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.170995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.171563 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.171065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-multus\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.171563 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.171081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-host\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.171563 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.171094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:10.203735 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.203562    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:10.203852 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.203833    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-65pss\""
Nov 03 11:50:10.205686 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.205671    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:10.205686 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.205673    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxdwl\""
Nov 03 11:50:10.207943 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.207925    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jp4k\""
Nov 03 11:50:10.208031 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.207988    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 11:50:10.209689 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.209668    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 11:50:10.209689 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.209679    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 11:50:10.209689 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.209690    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 11:50:10.209870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.209738    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 11:50:10.209870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.209746    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 11:50:10.209870 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.209782    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 11:50:10.213525 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.213512    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 11:50:10.215498 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.215481    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 11:50:10.271674 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-kubelet-config\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.271787 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.271787 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-socket-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.271787 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271758    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-kubelet-config\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.271879 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktl5\" (UniqueName: \"kubernetes.io/projected/9ceda7ca-9770-42ec-973a-0435ff74dd29-kube-api-access-lktl5\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.271879 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysconfig\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.271879 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-var-lib-kubelet\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.271879 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271862    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-socket-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.272021 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-systemd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272021 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-netd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272021 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-system-cni-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.272021 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271952    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-var-lib-kubelet\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.272021 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.271959    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:10.272021 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271986    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-systemd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.271943    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysconfig\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272051    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-netd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.272100    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret podName:098926dd-2788-41f4-a964-5f7448c8c41f nodeName:}" failed. No retries permitted until 2026-11-03 11:50:10.772067181 +0000 UTC m=+2.292561280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret") pod "global-pull-secret-syncer-lsvz7" (UID: "098926dd-2788-41f4-a964-5f7448c8c41f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272100    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-system-cni-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272128    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-multus\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-multus\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-host\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cnibin\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.272210 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-hostroot\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272256    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cnibin\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272263    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-hostroot\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272263    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-host\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.272306    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.272337    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:10.772325687 +0000 UTC m=+2.292819812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-multus-certs\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272366    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-multus-certs\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlbr\" (UniqueName: \"kubernetes.io/projected/bfd61052-ba77-4b91-bb1b-9d808791b96b-kube-api-access-smlbr\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-systemd-units\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-netns\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272464    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-systemd-units\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdzg\" (UniqueName: \"kubernetes.io/projected/874b36e5-1db3-498f-a51c-e23ccfb5947b-kube-api-access-ccdzg\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272491    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-netns\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-sys-fs\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.272544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272528    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272583    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-sys-fs\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9t6r\" (UniqueName: \"kubernetes.io/projected/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-kube-api-access-j9t6r\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272606    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-system-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-etc-kubernetes\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272689    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-system-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-conf\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272731    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-etc-kubernetes\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-systemd\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272769    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-script-lib\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-registration-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272837    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-conf\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272842    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-systemd\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d378c41e-4f5f-4f08-b051-23694fee59d4-host\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.273194 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272874    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-registration-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-bin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272917    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d378c41e-4f5f-4f08-b051-23694fee59d4-host\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-conf-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-run\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272976    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-bin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.272981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-host-slash\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273026    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-conf-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d378c41e-4f5f-4f08-b051-23694fee59d4-serviceca\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-netns\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273093    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-run\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273046    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-host-slash\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/77d106d7-59a7-459c-b012-aaa087cb50d9-konnectivity-ca\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273132    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-netns\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-node-log\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273159    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-node-log\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq6q\" (UniqueName: \"kubernetes.io/projected/d378c41e-4f5f-4f08-b051-23694fee59d4-kube-api-access-nnq6q\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.274035 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-kubernetes\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-log-socket\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovn-node-metrics-cert\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-device-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqw5\" (UniqueName: \"kubernetes.io/projected/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-kube-api-access-xkqw5\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273292    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-log-socket\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-k8s-cni-cncf-io\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273266    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-kubernetes\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-tmp\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273353    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-k8s-cni-cncf-io\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273358    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-device-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-iptables-alerter-script\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-slash\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-env-overrides\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273469    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-script-lib\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.274835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d378c41e-4f5f-4f08-b051-23694fee59d4-serviceca\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273550    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-slash\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-hosts-file\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273581    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-daemon-config\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/77d106d7-59a7-459c-b012-aaa087cb50d9-agent-certs\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-sys\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-config\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273718    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/77d106d7-59a7-459c-b012-aaa087cb50d9-konnectivity-ca\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273781    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273932    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-sys\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274000    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-iptables-alerter-script\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274073    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.273613    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274205    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-hosts-file\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.275450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxwp\" (UniqueName: \"kubernetes.io/projected/b9319607-a6c8-4003-89bd-14da12ba17a8-kube-api-access-hzxwp\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274284    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274337    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-env-overrides\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274345    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-socket-dir-parent\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274348    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-config\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-modprobe-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-kubelet\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274448    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-var-lib-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274526    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-modprobe-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274518    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-socket-dir-parent\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-etc-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274560    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-var-lib-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274575    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-kubelet\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-ovn\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274633    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-ovn\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-tmp-dir\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-kubelet\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274616    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-etc-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-lib-modules\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-tuned\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274744    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-kubelet\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-dbus\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-cnibin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxpx9\" (UniqueName: \"kubernetes.io/projected/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-kube-api-access-zxpx9\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mn4m\" (UniqueName: \"kubernetes.io/projected/a208df1b-dba2-4b61-bd72-d7a4dffcada0-kube-api-access-5mn4m\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-cni-binary-copy\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-bin\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-dbus\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.275036    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-bin\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.275105    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.275131    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-cnibin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.276569 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.274814    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-lib-modules\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.277234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.276333    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-daemon-config\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.277234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.276389    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-tmp-dir\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.277234 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.276509    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-cni-binary-copy\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.277371 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.277290    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-tmp\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.277371 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.277361    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovn-node-metrics-cert\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.277538 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.277512    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/77d106d7-59a7-459c-b012-aaa087cb50d9-agent-certs\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.278225 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.278206    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-tuned\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.333606 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.333548    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:10.333606 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.333569    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:10.333606 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.333582    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g9pcf for pod openshift-network-diagnostics/network-check-target-gkt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:10.333815 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.333649    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf podName:ad5bba67-4cce-447f-aa79-d6061b83fb63 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:10.833622902 +0000 UTC m=+2.354117014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g9pcf" (UniqueName: "kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf") pod "network-check-target-gkt2k" (UID: "ad5bba67-4cce-447f-aa79-d6061b83fb63") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:10.336311 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.335987    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq6q\" (UniqueName: \"kubernetes.io/projected/d378c41e-4f5f-4f08-b051-23694fee59d4-kube-api-access-nnq6q\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.336413 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.336370    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mn4m\" (UniqueName: \"kubernetes.io/projected/a208df1b-dba2-4b61-bd72-d7a4dffcada0-kube-api-access-5mn4m\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.336872 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.336852    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdzg\" (UniqueName: \"kubernetes.io/projected/874b36e5-1db3-498f-a51c-e23ccfb5947b-kube-api-access-ccdzg\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.336947 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.336863    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxwp\" (UniqueName: \"kubernetes.io/projected/b9319607-a6c8-4003-89bd-14da12ba17a8-kube-api-access-hzxwp\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.337134 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.337117    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9t6r\" (UniqueName: \"kubernetes.io/projected/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-kube-api-access-j9t6r\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.337233 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.337213    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxpx9\" (UniqueName: \"kubernetes.io/projected/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-kube-api-access-zxpx9\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:10.337346 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.337330    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqw5\" (UniqueName: \"kubernetes.io/projected/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-kube-api-access-xkqw5\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.337516 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.337502    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktl5\" (UniqueName: \"kubernetes.io/projected/9ceda7ca-9770-42ec-973a-0435ff74dd29-kube-api-access-lktl5\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.343978 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.343962    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlbr\" (UniqueName: \"kubernetes.io/projected/bfd61052-ba77-4b91-bb1b-9d808791b96b-kube-api-access-smlbr\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.435787 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.435765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:10.468812 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.468787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:10.469832 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.469801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dqh56"
Nov 03 11:50:10.497772 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.497752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:10.499969 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.499951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:10.508431 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.508415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:10.528457 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.528439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:10.530582 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.530564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:10.532438 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.532423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:10.569675 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:10.569646    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2905ce8bf73baad4434061659facba8.slice/crio-3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596 WatchSource:0}: Error finding container 3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596: Status 404 returned error can't find the container with id 3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596
Nov 03 11:50:10.570116 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:10.570096    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb495a64a0efaf6f05f21f42ec740163f.slice/crio-b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0 WatchSource:0}: Error finding container b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0: Status 404 returned error can't find the container with id b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0
Nov 03 11:50:10.575949 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.575932    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:50:10.779465 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.779396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:10.779465 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.779431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:10.779618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.779527    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:10.779618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.779534    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:10.779618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.779584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:11.779566308 +0000 UTC m=+3.300060402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:10.779618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.779599    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret podName:098926dd-2788-41f4-a964-5f7448c8c41f nodeName:}" failed. No retries permitted until 2026-11-03 11:50:11.779592434 +0000 UTC m=+3.300086525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret") pod "global-pull-secret-syncer-lsvz7" (UID: "098926dd-2788-41f4-a964-5f7448c8c41f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:10.840515 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.840494    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:10.872868 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.872848    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:10.880090 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:10.880071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:10.880175 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.880163    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:10.880249 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.880179    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:10.880249 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.880187    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g9pcf for pod openshift-network-diagnostics/network-check-target-gkt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:10.880249 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:10.880225    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf podName:ad5bba67-4cce-447f-aa79-d6061b83fb63 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:11.880214084 +0000 UTC m=+3.400708176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9pcf" (UniqueName: "kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf") pod "network-check-target-gkt2k" (UID: "ad5bba67-4cce-447f-aa79-d6061b83fb63") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:11.002208 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.002165    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"f2905ce8bf73baad4434061659facba8","Type":"ContainerStarted","Data":"3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596"}
Nov 03 11:50:11.003173 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.003151    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"b495a64a0efaf6f05f21f42ec740163f","Type":"ContainerStarted","Data":"b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0"}
Nov 03 11:50:11.155994 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.155917    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:09 +0000 UTC" deadline="2028-08-20 09:49:25.786270082 +0000 UTC"
Nov 03 11:50:11.155994 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.155953    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15741h59m14.63032033s"
Nov 03 11:50:11.220611 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.220592    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:11.316327 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.316302    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ceda7ca_9770_42ec_973a_0435ff74dd29.slice/crio-9351f15d426761591aede7a3eea7c78302a5453e1130a2a906aa95359236d0e3 WatchSource:0}: Error finding container 9351f15d426761591aede7a3eea7c78302a5453e1130a2a906aa95359236d0e3: Status 404 returned error can't find the container with id 9351f15d426761591aede7a3eea7c78302a5453e1130a2a906aa95359236d0e3
Nov 03 11:50:11.347335 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.347298    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874b36e5_1db3_498f_a51c_e23ccfb5947b.slice/crio-8dd2599074cd01041c6bec5f9e8a8d53b577f2c5d24cf29588e54c9ef9a13c9d WatchSource:0}: Error finding container 8dd2599074cd01041c6bec5f9e8a8d53b577f2c5d24cf29588e54c9ef9a13c9d: Status 404 returned error can't find the container with id 8dd2599074cd01041c6bec5f9e8a8d53b577f2c5d24cf29588e54c9ef9a13c9d
Nov 03 11:50:11.375992 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.375971    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daea77e_b365_4fc5_a5f3_bd9ad7c615e6.slice/crio-281628f75b067a433c06ff16bdd0c7a98ab29c79d922f092875c0a049878eef3 WatchSource:0}: Error finding container 281628f75b067a433c06ff16bdd0c7a98ab29c79d922f092875c0a049878eef3: Status 404 returned error can't find the container with id 281628f75b067a433c06ff16bdd0c7a98ab29c79d922f092875c0a049878eef3
Nov 03 11:50:11.386054 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.386033    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda208df1b_dba2_4b61_bd72_d7a4dffcada0.slice/crio-adb334b78165ad565da8592a1ac6ce4e420f2cd162cdd1350924cd26e1f645a4 WatchSource:0}: Error finding container adb334b78165ad565da8592a1ac6ce4e420f2cd162cdd1350924cd26e1f645a4: Status 404 returned error can't find the container with id adb334b78165ad565da8592a1ac6ce4e420f2cd162cdd1350924cd26e1f645a4
Nov 03 11:50:11.393363 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.393332    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd61052_ba77_4b91_bb1b_9d808791b96b.slice/crio-782c7dc68e93b1f67464b799dbcc9f21818fcf1a09bafbda90a1dde77b9314bd WatchSource:0}: Error finding container 782c7dc68e93b1f67464b799dbcc9f21818fcf1a09bafbda90a1dde77b9314bd: Status 404 returned error can't find the container with id 782c7dc68e93b1f67464b799dbcc9f21818fcf1a09bafbda90a1dde77b9314bd
Nov 03 11:50:11.394225 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.394206    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9319607_a6c8_4003_89bd_14da12ba17a8.slice/crio-254ca63a2723b5c0fe1ccae51c7c0f13eed703349d08bc8ab2a278059788e93b WatchSource:0}: Error finding container 254ca63a2723b5c0fe1ccae51c7c0f13eed703349d08bc8ab2a278059788e93b: Status 404 returned error can't find the container with id 254ca63a2723b5c0fe1ccae51c7c0f13eed703349d08bc8ab2a278059788e93b
Nov 03 11:50:11.395175 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.395144    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7f0718_4ffa_4b52_abfd_0666d8a6cfb2.slice/crio-2bb466f00cd21ebce72be2d5610ab182cb6b11efe172907f3d66976dd5664c99 WatchSource:0}: Error finding container 2bb466f00cd21ebce72be2d5610ab182cb6b11efe172907f3d66976dd5664c99: Status 404 returned error can't find the container with id 2bb466f00cd21ebce72be2d5610ab182cb6b11efe172907f3d66976dd5664c99
Nov 03 11:50:11.396209 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.396189    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd378c41e_4f5f_4f08_b051_23694fee59d4.slice/crio-63d24c65779b165b896ea304f0f73add40f26a0462206503242dd5944c25a7a2 WatchSource:0}: Error finding container 63d24c65779b165b896ea304f0f73add40f26a0462206503242dd5944c25a7a2: Status 404 returned error can't find the container with id 63d24c65779b165b896ea304f0f73add40f26a0462206503242dd5944c25a7a2
Nov 03 11:50:11.464295 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:11.464277    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d106d7_59a7_459c_b012_aaa087cb50d9.slice/crio-61bdc96b360e4d56ebadaade76a91acc2452433bd30d184dd48e27ddfcc00593 WatchSource:0}: Error finding container 61bdc96b360e4d56ebadaade76a91acc2452433bd30d184dd48e27ddfcc00593: Status 404 returned error can't find the container with id 61bdc96b360e4d56ebadaade76a91acc2452433bd30d184dd48e27ddfcc00593
Nov 03 11:50:11.785862 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.785784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:11.785862 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.785833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:11.786074 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.785959    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:11.786074 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.785960    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:11.786074 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.786015    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:13.785998487 +0000 UTC m=+5.306492583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:11.786074 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.786033    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret podName:098926dd-2788-41f4-a964-5f7448c8c41f nodeName:}" failed. No retries permitted until 2026-11-03 11:50:13.786024252 +0000 UTC m=+5.306518345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret") pod "global-pull-secret-syncer-lsvz7" (UID: "098926dd-2788-41f4-a964-5f7448c8c41f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:11.886890 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.886855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:11.887432 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.887039    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:11.887432 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.887055    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:11.887432 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.887066    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g9pcf for pod openshift-network-diagnostics/network-check-target-gkt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:11.887432 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.887118    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf podName:ad5bba67-4cce-447f-aa79-d6061b83fb63 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:13.887101131 +0000 UTC m=+5.407595223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9pcf" (UniqueName: "kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf") pod "network-check-target-gkt2k" (UID: "ad5bba67-4cce-447f-aa79-d6061b83fb63") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:11.999587 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.999550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:11.999587 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.999585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:11.999828 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:11.999585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:11.999828 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.999686    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:11.999828 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.999784    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:11.999980 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:11.999874    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:12.005901 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.005869    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-df7t5" event={"ID":"77d106d7-59a7-459c-b012-aaa087cb50d9","Type":"ContainerStarted","Data":"61bdc96b360e4d56ebadaade76a91acc2452433bd30d184dd48e27ddfcc00593"}
Nov 03 11:50:12.007175 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.007146    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7bdv2" event={"ID":"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2","Type":"ContainerStarted","Data":"2bb466f00cd21ebce72be2d5610ab182cb6b11efe172907f3d66976dd5664c99"}
Nov 03 11:50:12.008470 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.008443    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-742vk" event={"ID":"bfd61052-ba77-4b91-bb1b-9d808791b96b","Type":"ContainerStarted","Data":"782c7dc68e93b1f67464b799dbcc9f21818fcf1a09bafbda90a1dde77b9314bd"}
Nov 03 11:50:12.009749 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.009727    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg" event={"ID":"b9319607-a6c8-4003-89bd-14da12ba17a8","Type":"ContainerStarted","Data":"254ca63a2723b5c0fe1ccae51c7c0f13eed703349d08bc8ab2a278059788e93b"}
Nov 03 11:50:12.011045 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.011026    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8pdls" event={"ID":"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6","Type":"ContainerStarted","Data":"281628f75b067a433c06ff16bdd0c7a98ab29c79d922f092875c0a049878eef3"}
Nov 03 11:50:12.012356 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.012331    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcljv" event={"ID":"d378c41e-4f5f-4f08-b051-23694fee59d4","Type":"ContainerStarted","Data":"63d24c65779b165b896ea304f0f73add40f26a0462206503242dd5944c25a7a2"}
Nov 03 11:50:12.013744 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.013724    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerStarted","Data":"adb334b78165ad565da8592a1ac6ce4e420f2cd162cdd1350924cd26e1f645a4"}
Nov 03 11:50:12.015029 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.014967    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"8dd2599074cd01041c6bec5f9e8a8d53b577f2c5d24cf29588e54c9ef9a13c9d"}
Nov 03 11:50:12.016238 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.016216    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dqh56" event={"ID":"9ceda7ca-9770-42ec-973a-0435ff74dd29","Type":"ContainerStarted","Data":"9351f15d426761591aede7a3eea7c78302a5453e1130a2a906aa95359236d0e3"}
Nov 03 11:50:12.156886 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.156782    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:09 +0000 UTC" deadline="2028-05-03 11:21:58.734526531 +0000 UTC"
Nov 03 11:50:12.156886 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:12.156829    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13127h31m46.577700503s"
Nov 03 11:50:13.809723 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:13.809459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:13.810195 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:13.809766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:13.810195 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.810007    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:13.810195 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.810060    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:17.810042043 +0000 UTC m=+9.330536151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:13.810195 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.810149    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:13.810195 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.810185    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret podName:098926dd-2788-41f4-a964-5f7448c8c41f nodeName:}" failed. No retries permitted until 2026-11-03 11:50:17.810173817 +0000 UTC m=+9.330667924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret") pod "global-pull-secret-syncer-lsvz7" (UID: "098926dd-2788-41f4-a964-5f7448c8c41f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:13.911082 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:13.911047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:13.911230 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.911203    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:13.911230 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.911226    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:13.911469 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.911239    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g9pcf for pod openshift-network-diagnostics/network-check-target-gkt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:13.911469 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.911306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf podName:ad5bba67-4cce-447f-aa79-d6061b83fb63 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:17.91128147 +0000 UTC m=+9.431775563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9pcf" (UniqueName: "kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf") pod "network-check-target-gkt2k" (UID: "ad5bba67-4cce-447f-aa79-d6061b83fb63") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:13.999059 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:13.998943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:13.999254 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.999076    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:13.999254 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:13.999140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:13.999254 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.999221    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:13.999430 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:13.999259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:13.999430 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:13.999337    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:14.032605 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:14.032570    2581 generic.go:356] "Generic (PLEG): container finished" podID="b495a64a0efaf6f05f21f42ec740163f" containerID="af2a3b42bc6fbf04a935dcba5ed69d97ae2f69eea46c2aa86584475750122cd9" exitCode=0
Nov 03 11:50:14.032783 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:14.032634    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"b495a64a0efaf6f05f21f42ec740163f","Type":"ContainerDied","Data":"af2a3b42bc6fbf04a935dcba5ed69d97ae2f69eea46c2aa86584475750122cd9"}
Nov 03 11:50:14.043782 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:14.043758    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"f2905ce8bf73baad4434061659facba8","Type":"ContainerStarted","Data":"3a22d380c8920eaf8c80fc2ac371118649209bb665d3855a43f72ee0db7e6305"}
Nov 03 11:50:15.064091 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.064050    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"b495a64a0efaf6f05f21f42ec740163f","Type":"ContainerStarted","Data":"d791d084baefd4e028ae33c54dc6e0c8573fb3ef8ed4798c095fa083598bf749"}
Nov 03 11:50:15.070536 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.070507    2581 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="95d471dc74808ef65b660e60af940711015f9064bae89e67cd10e49e01fc1828" exitCode=0
Nov 03 11:50:15.071397 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.071375    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerDied","Data":"95d471dc74808ef65b660e60af940711015f9064bae89e67cd10e49e01fc1828"}
Nov 03 11:50:15.178176 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.176955    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal" podStartSLOduration=6.176935517 podStartE2EDuration="6.176935517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:14.260814869 +0000 UTC m=+5.781308990" watchObservedRunningTime="2026-11-03 11:50:15.176935517 +0000 UTC m=+6.697429632"
Nov 03 11:50:15.295864 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.294608    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal" podStartSLOduration=6.294588864 podStartE2EDuration="6.294588864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:15.178006673 +0000 UTC m=+6.698500788" watchObservedRunningTime="2026-11-03 11:50:15.294588864 +0000 UTC m=+6.815082969"
Nov 03 11:50:15.999154 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.999127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:15.999305 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:15.999242    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:15.999368 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.999328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:15.999465 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:15.999419    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:15.999524 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:15.999476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:15.999571 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:15.999544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:16.083080 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:16.082975    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-df7t5" event={"ID":"77d106d7-59a7-459c-b012-aaa087cb50d9","Type":"ContainerStarted","Data":"1187ba08edb8fbf0dbded89364c598febdb7dc86ab3b68c7772bd9103b979018"}
Nov 03 11:50:17.694629 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:17.694603    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:17.843443 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:17.843412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:17.843606 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:17.843452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:17.843606 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.843581    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:17.843733 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.843631    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:25.84361955 +0000 UTC m=+17.364113654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:17.843733 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.843581    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:17.843733 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.843670    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret podName:098926dd-2788-41f4-a964-5f7448c8c41f nodeName:}" failed. No retries permitted until 2026-11-03 11:50:25.843660827 +0000 UTC m=+17.364154939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret") pod "global-pull-secret-syncer-lsvz7" (UID: "098926dd-2788-41f4-a964-5f7448c8c41f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:17.944619 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:17.944187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:17.944619 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.944440    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:17.944619 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.944464    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:17.944619 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.944477    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g9pcf for pod openshift-network-diagnostics/network-check-target-gkt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:17.944619 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.944528    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf podName:ad5bba67-4cce-447f-aa79-d6061b83fb63 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:25.944512845 +0000 UTC m=+17.465006939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9pcf" (UniqueName: "kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf") pod "network-check-target-gkt2k" (UID: "ad5bba67-4cce-447f-aa79-d6061b83fb63") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:17.998922 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:17.998884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:17.999107 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.999017    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:17.999107 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:17.999088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:17.999209 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.999159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:17.999209 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:17.999206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:17.999362 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:17.999268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:18.093844 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:18.093171    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcljv" event={"ID":"d378c41e-4f5f-4f08-b051-23694fee59d4","Type":"ContainerStarted","Data":"81eb801004b0ebb206e40c75cdd51b88bd5f42c17cb2c0e9d9a37cd72bbeea55"}
Nov 03 11:50:18.096182 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:18.096103    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg" event={"ID":"b9319607-a6c8-4003-89bd-14da12ba17a8","Type":"ContainerStarted","Data":"be78cc89bc22938eae57deebe0e5c18c8fa3b73fcd7a9a23c7b16e39a1476f47"}
Nov 03 11:50:18.211708 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:18.210856    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-df7t5" podStartSLOduration=4.847950982 podStartE2EDuration="9.210836453s" totalImagesPullingTime="4.362885471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:16.197017599 +0000 UTC m=+7.717511713" watchObservedRunningTime="2026-11-03 11:50:18.210836453 +0000 UTC m=+9.731330568"
Nov 03 11:50:19.834440 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:19.833761    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 11:50:19.919489 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:19.919378    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T11:50:19.833805493Z","UUID":"6bae02db-414c-4079-99e7-44ff09004726","Handler":null,"Name":"","Endpoint":""}
Nov 03 11:50:19.922247 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:19.922227    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 11:50:19.922345 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:19.922259    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 11:50:19.999120 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:19.999056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:19.999230 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:19.999150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:19.999230 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:19.999168    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:19.999319 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:19.999255    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:19.999365 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:19.999347    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:19.999423 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:19.999410    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:20.102089 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.101861    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7bdv2" event={"ID":"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2","Type":"ContainerStarted","Data":"b33a54f7519a9737f57f4f25933e9bb25ff91a687fbdf1516fc5428ee26be870"}
Nov 03 11:50:20.104092 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.104068    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg" event={"ID":"b9319607-a6c8-4003-89bd-14da12ba17a8","Type":"ContainerStarted","Data":"92c647ca94f70d2511a67752f3d66ec93d50bb87f346e0275479efa2e40b306c"}
Nov 03 11:50:20.105842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.105820    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dqh56" event={"ID":"9ceda7ca-9770-42ec-973a-0435ff74dd29","Type":"ContainerStarted","Data":"bb1145e1bb5bd4b54eea6edd02c7244a94bf2f9f2af0239063ee2d19fa049b1c"}
Nov 03 11:50:20.219006 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.218902    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dcljv" podStartSLOduration=5.676388853 podStartE2EDuration="11.218883396s" totalImagesPullingTime="5.542494543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:18.211632265 +0000 UTC m=+9.732126380" watchObservedRunningTime="2026-11-03 11:50:20.218883396 +0000 UTC m=+11.739377511"
Nov 03 11:50:20.219745 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.219281    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7bdv2" podStartSLOduration=2.803225054 podStartE2EDuration="11.219271049s" totalImagesPullingTime="8.416045995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:20.218538996 +0000 UTC m=+11.739033110" watchObservedRunningTime="2026-11-03 11:50:20.219271049 +0000 UTC m=+11.739765163"
Nov 03 11:50:20.320368 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.320339    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:20.321147 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.321125    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:20.337894 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:20.337853    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dqh56" podStartSLOduration=3.2166168490000002 podStartE2EDuration="11.337842466s" totalImagesPullingTime="8.121225617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:20.337250461 +0000 UTC m=+11.857744576" watchObservedRunningTime="2026-11-03 11:50:20.337842466 +0000 UTC m=+11.858336580"
Nov 03 11:50:21.109829 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.109790    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8pdls" event={"ID":"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6","Type":"ContainerStarted","Data":"dcf0b175ac152d11f7be38cc17286d0b383f836cdd6377a0713d7a84c6d7070f"}
Nov 03 11:50:21.112292 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.112253    2581 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="af15450a49a195894df480790c87b85a88c0e8b8f705ff654a3adaa9ed5fbb83" exitCode=0
Nov 03 11:50:21.112808 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.112776    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerDied","Data":"af15450a49a195894df480790c87b85a88c0e8b8f705ff654a3adaa9ed5fbb83"}
Nov 03 11:50:21.113420 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.113398    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:21.224107 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.224058    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8pdls" podStartSLOduration=4.074551877 podStartE2EDuration="12.224039484s" totalImagesPullingTime="8.149487607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:21.223733282 +0000 UTC m=+12.744227398" watchObservedRunningTime="2026-11-03 11:50:21.224039484 +0000 UTC m=+12.744533598"
Nov 03 11:50:21.999014 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.998990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:21.999126 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.999086    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:21.999126 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:21.999102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:21.999232 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:21.999181    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:21.999291 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:21.999230    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:21.999351 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:21.999312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:22.116195 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:22.116162    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-742vk" event={"ID":"bfd61052-ba77-4b91-bb1b-9d808791b96b","Type":"ContainerStarted","Data":"2517e7476256dad9be4d01447e99cc5b71cd2a652105b2e6b04857d97f2b3efe"}
Nov 03 11:50:22.121066 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:22.121036    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg" event={"ID":"b9319607-a6c8-4003-89bd-14da12ba17a8","Type":"ContainerStarted","Data":"3ee032e379dd8fc5310d7d77602a98f121fcd8000710a078cc14ffa028f582b7"}
Nov 03 11:50:22.231443 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:22.231349    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-742vk" podStartSLOduration=2.725947517 podStartE2EDuration="13.231335339s" totalImagesPullingTime="10.505387822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:22.23087717 +0000 UTC m=+13.751371285" watchObservedRunningTime="2026-11-03 11:50:22.231335339 +0000 UTC m=+13.751829452"
Nov 03 11:50:22.348315 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:22.348267    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg" podStartSLOduration=3.926848517 podStartE2EDuration="13.348248695s" totalImagesPullingTime="9.421400178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:22.34767144 +0000 UTC m=+13.868165555" watchObservedRunningTime="2026-11-03 11:50:22.348248695 +0000 UTC m=+13.868742809"
Nov 03 11:50:23.124858 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:23.124826    2581 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="9fcfb324eacf82d2cb783f32e95f4b67ab76d21b00a513c241c833626302cd4e" exitCode=0
Nov 03 11:50:23.125846 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:23.125826    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerDied","Data":"9fcfb324eacf82d2cb783f32e95f4b67ab76d21b00a513c241c833626302cd4e"}
Nov 03 11:50:23.999826 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:23.999636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:24.000048 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:23.999633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:24.000048 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:23.999905    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:24.000048 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:24.000009    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:24.000048 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:23.999651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:24.000224 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:24.000116    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:25.131308 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:25.131265    2581 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="4c7a9dc7cd00fbb38aca0e093df6c44a41a19b790216889dbd2e06f37b8545fb" exitCode=0
Nov 03 11:50:25.131790 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:25.131322    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerDied","Data":"4c7a9dc7cd00fbb38aca0e093df6c44a41a19b790216889dbd2e06f37b8545fb"}
Nov 03 11:50:25.904442 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:25.904410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:25.904442 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:25.904448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:25.904618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:25.904543    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:25.904618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:25.904544    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:25.904618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:25.904588    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.904575657 +0000 UTC m=+33.425069748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:25.904618 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:25.904600    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret podName:098926dd-2788-41f4-a964-5f7448c8c41f nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.904594357 +0000 UTC m=+33.425088448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret") pod "global-pull-secret-syncer-lsvz7" (UID: "098926dd-2788-41f4-a964-5f7448c8c41f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:25.999665 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:25.999634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:25.999822 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:25.999669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:25.999822 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:25.999742    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:25.999822 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:25.999784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:25.999968 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:25.999847    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:25.999968 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:25.999921    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:26.004797 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:26.004778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:26.004890 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:26.004877    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:26.004890 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:26.004889    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:26.004957 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:26.004898    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g9pcf for pod openshift-network-diagnostics/network-check-target-gkt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:26.004957 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:26.004941    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf podName:ad5bba67-4cce-447f-aa79-d6061b83fb63 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:42.0049298 +0000 UTC m=+33.525423892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9pcf" (UniqueName: "kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf") pod "network-check-target-gkt2k" (UID: "ad5bba67-4cce-447f-aa79-d6061b83fb63") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:27.139064 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.139025    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"a785a7f2e9d2fbd971fdc38e8d483ad5c122d9e4fae24ae6614e3e91b85a0d21"}
Nov 03 11:50:27.139541 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.139107    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"a3ad52a57f2dcdd1e8b9a028314332215f386b8e4765d163c47e2c92433ce6f6"}
Nov 03 11:50:27.139541 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.139147    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"75f6c2910e690476ef9108e5025d2e47fd78705e81bc3f890a943fd02feeb181"}
Nov 03 11:50:27.139541 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.139175    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"62e9464bca603a8ca0cdfb043b41ba797e24b5e6884be60cda9eab66fcabb0e7"}
Nov 03 11:50:27.139541 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.139206    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"b87f3919f7b07dc576cb50e767796e76cacc1f0676481c5530e956a892e4023d"}
Nov 03 11:50:27.139541 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.139247    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"8adf3ca200b7f8305f1352bdc9282b03981fd599af25f048d4d86555288f0234"}
Nov 03 11:50:27.999316 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.999283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:27.999316 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.999309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:27.999510 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:27.999285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:27.999510 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:27.999372    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:27.999510 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:27.999451    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:27.999510 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:27.999502    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:28.143652 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:28.143629    2581 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="736cd7fd5bd1bba313b7996f2fd3372a94446e7b49d3e644ccdb8aa23b390417" exitCode=0
Nov 03 11:50:28.144075 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:28.143666    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerDied","Data":"736cd7fd5bd1bba313b7996f2fd3372a94446e7b49d3e644ccdb8aa23b390417"}
Nov 03 11:50:29.148824 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:29.148733    2581 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="1a74c97683e5e5ca1aa9360db74296aa65de9f97127622f07482902310e3d9b6" exitCode=0
Nov 03 11:50:29.148824 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:29.148810    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerDied","Data":"1a74c97683e5e5ca1aa9360db74296aa65de9f97127622f07482902310e3d9b6"}
Nov 03 11:50:29.151939 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:29.151920    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"ca6dd8a653f00b55d15265d2dde5b2f1782ec8f8073b27db613b1730561d63b3"}
Nov 03 11:50:29.998909 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:29.998879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:29.998909 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:29.998908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:29.999103 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:29.998985    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:29.999103 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:29.999023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:29.999103 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:29.999093    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:29.999190 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:29.999157    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:30.156363 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:30.156334    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" event={"ID":"a208df1b-dba2-4b61-bd72-d7a4dffcada0","Type":"ContainerStarted","Data":"44724880e4727dbb8f97f862ea0df7ed9073771418c85f1f1755603ae932b9f8"}
Nov 03 11:50:30.274538 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:30.274455    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j7bdh" podStartSLOduration=8.646553129 podStartE2EDuration="21.274442066s" totalImagesPullingTime="12.627888937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:30.273936413 +0000 UTC m=+21.794430527" watchObservedRunningTime="2026-11-03 11:50:30.274442066 +0000 UTC m=+21.794936179"
Nov 03 11:50:31.161564 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:31.161329    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" event={"ID":"874b36e5-1db3-498f-a51c-e23ccfb5947b","Type":"ContainerStarted","Data":"d37453797684776600f18dad62a7a2c216f3d1107f24a7e73444e026f3ac5f57"}
Nov 03 11:50:31.384799 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:31.384748    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7" podStartSLOduration=7.465362357 podStartE2EDuration="22.384736256s" totalImagesPullingTime="14.919373899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:31.384203213 +0000 UTC m=+22.904697326" watchObservedRunningTime="2026-11-03 11:50:31.384736256 +0000 UTC m=+22.905230370"
Nov 03 11:50:31.999428 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:31.999392    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:31.999428 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:31.999416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:31.999791 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:31.999537    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:31.999791 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:31.999588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:31.999791 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:31.999717    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:31.999791 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:31.999780    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:32.163947 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:32.163820    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:32.163947 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:32.163906    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:32.163947 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:32.163953    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:32.179206 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:32.179186    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:32.179391 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:32.179335    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:33.488130 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:33.488102    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkt2k"]
Nov 03 11:50:33.488575 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:33.488224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:33.488575 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:33.488309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:33.491066 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:33.491045    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cs8vd"]
Nov 03 11:50:33.491191 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:33.491126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:33.491253 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:33.491204    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:33.491614 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:33.491594    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsvz7"]
Nov 03 11:50:33.491717 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:33.491691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:33.491817 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:33.491798    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:34.998659 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:34.998628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:34.998659 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:34.998646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:34.999042 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:34.998627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:34.999042 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:34.998750    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:34.999042 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:34.998826    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:34.999042 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:34.998917    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:36.999713 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:36.999664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:37.000231 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:36.999671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:37.000231 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:36.999803    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:37.000231 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:36.999870    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:37.000231 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:36.999904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:37.000231 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:37.000024    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:39.000189 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.000160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:39.000533 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.000245    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cs8vd" podUID="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1"
Nov 03 11:50:39.000533 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.000256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:39.000533 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.000287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:39.000533 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.000338    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2k" podUID="ad5bba67-4cce-447f-aa79-d6061b83fb63"
Nov 03 11:50:39.000533 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.000415    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsvz7" podUID="098926dd-2788-41f4-a964-5f7448c8c41f"
Nov 03 11:50:39.270862 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.270788    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeReady"
Nov 03 11:50:39.270995 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.270900    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 11:50:39.425601 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.425573    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nct5z"]
Nov 03 11:50:39.431025 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.431003    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n7zd8"]
Nov 03 11:50:39.431141 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.431124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nct5z"
Nov 03 11:50:39.434188 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.434167    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xv5pl"]
Nov 03 11:50:39.434348 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.434284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-n7zd8"
Nov 03 11:50:39.444094 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.443907    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6796696b5b-mlp66"]
Nov 03 11:50:39.444798 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.444776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xv5pl"
Nov 03 11:50:39.447891 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.447872    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"]
Nov 03 11:50:39.448010 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.447993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.452606 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.452583    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8cwrc"]
Nov 03 11:50:39.452713 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.452661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.454779 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.454763    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"]
Nov 03 11:50:39.455445 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.455428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.459529 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.459508    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"]
Nov 03 11:50:39.460547 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.459731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.462208 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.462186    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"]
Nov 03 11:50:39.462596 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.462579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.468295 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.468033    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b5bc54f87-gkx8h"]
Nov 03 11:50:39.468426 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.468407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:39.471800 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.471783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.476607 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.476592    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"]
Nov 03 11:50:39.479715 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.479679    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b9dcd8545-gs4r8"]
Nov 03 11:50:39.479802 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.479777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.483218 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.483201    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"]
Nov 03 11:50:39.483335 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.483316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.486342 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.486326    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"]
Nov 03 11:50:39.486480 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.486459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.489199 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.489169    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"]
Nov 03 11:50:39.489378 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.489355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"
Nov 03 11:50:39.497950 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.497932    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w7l5k"]
Nov 03 11:50:39.498073 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.498057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"
Nov 03 11:50:39.501710 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.501680    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7j2qr"]
Nov 03 11:50:39.501799 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.501784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.505144 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.505118    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nct5z"]
Nov 03 11:50:39.505227 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.505184    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n7zd8"]
Nov 03 11:50:39.505227 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.505210    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xv5pl"]
Nov 03 11:50:39.505323 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.505241    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"]
Nov 03 11:50:39.505323 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.505262    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"]
Nov 03 11:50:39.507927 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.507903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.508032 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.507941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef83543a-8e80-486b-adeb-4d5aad83d4b0-serving-cert\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.508032 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.507969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gzp\" (UniqueName: \"kubernetes.io/projected/ef83543a-8e80-486b-adeb-4d5aad83d4b0-kube-api-access-z4gzp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.508144 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hsb\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.508144 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.508253 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfk8\" (UniqueName: \"kubernetes.io/projected/9e19744a-acf7-4e34-957f-c01efa20f0b2-kube-api-access-wkfk8\") pod \"kas-connection-checker-685895c7c9-nct5z\" (UID: \"9e19744a-acf7-4e34-957f-c01efa20f0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-nct5z"
Nov 03 11:50:39.508301 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.508301 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.508400 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c1607bc-d5e4-46b1-be5e-e27e7462a967-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.508400 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmlvw\" (UniqueName: \"kubernetes.io/projected/8e156d6b-a0b5-460d-8946-7f7329753e54-kube-api-access-gmlvw\") pod \"kas-connection-checker-685895c7c9-xv5pl\" (UID: \"8e156d6b-a0b5-460d-8946-7f7329753e54\") " pod="kube-system/kas-connection-checker-685895c7c9-xv5pl"
Nov 03 11:50:39.508505 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.508505 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bab06cb5-f005-4c43-a247-5f38134286cb-config\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.508505 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.508505 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxks\" (UniqueName: \"kubernetes.io/projected/4f115611-61eb-45bb-9e13-34b7a2519dd5-kube-api-access-bxxks\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.508505 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwz5v\" (UniqueName: \"kubernetes.io/projected/bab06cb5-f005-4c43-a247-5f38134286cb-kube-api-access-qwz5v\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.508734 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nf6r\" (UniqueName: \"kubernetes.io/projected/8c1607bc-d5e4-46b1-be5e-e27e7462a967-kube-api-access-7nf6r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.508734 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.508734 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bab06cb5-f005-4c43-a247-5f38134286cb-serving-cert\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.508734 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:39.508734 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-service-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.508969 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.508969 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-tmp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.508969 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-snapshots\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.508969 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c1607bc-d5e4-46b1-be5e-e27e7462a967-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.509153 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.508995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.509153 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkkf\" (UniqueName: \"kubernetes.io/projected/7b7eef80-4dd6-4087-b74c-66adfb323841-kube-api-access-xqkkf\") pod \"kas-connection-checker-685895c7c9-n7zd8\" (UID: \"7b7eef80-4dd6-4087-b74c-66adfb323841\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zd8"
Nov 03 11:50:39.509153 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.509457 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.505286    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6796696b5b-mlp66"]
Nov 03 11:50:39.509553 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509477    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"]
Nov 03 11:50:39.509553 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509503    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"]
Nov 03 11:50:39.509553 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509525    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"]
Nov 03 11:50:39.509553 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509546    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9dcd8545-gs4r8"]
Nov 03 11:50:39.509781 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509564    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"]
Nov 03 11:50:39.509781 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.509576    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7j2qr"]
Nov 03 11:50:39.526064 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.526039    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.526157 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.526079    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.526157 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.526121    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 11:50:39.526157 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.526118    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 11:50:39.526310 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.526187    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 11:50:39.527064 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.527046    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7l5k"]
Nov 03 11:50:39.527681 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.527663    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8cwrc"]
Nov 03 11:50:39.528304 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.528286    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b5bc54f87-gkx8h"]
Nov 03 11:50:39.534015 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.533998    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"]
Nov 03 11:50:39.534958 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.534941    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"]
Nov 03 11:50:39.584667 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.584641    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.584667 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.584647    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.584904 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.584883    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 11:50:39.584997 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.584933    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 11:50:39.585168 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585120    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585168 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585147    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585168 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585155    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 11:50:39.585168 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585152    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585339 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585152    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 11:50:39.585339 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585180    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585339 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585172    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585461 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585358    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 11:50:39.585461 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585386    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585559 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585489    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585559 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585533    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585559 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585536    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585559 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585552    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585723 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585555    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 11:50:39.585723 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585555    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585723 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585557    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585723 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585591    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585723 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585607    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585884 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585724    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.585884 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585740    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.585884 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585819    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 11:50:39.585884 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585844    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 11:50:39.586034 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585902    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:39.586034 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.585922    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:39.586170 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.586159    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 11:50:39.610341 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be887ffe-4891-4231-93ee-13d03c4efd85-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:39.610428 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-config\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.610428 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.610527 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.610527 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c1607bc-d5e4-46b1-be5e-e27e7462a967-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.610620 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7m9\" (UniqueName: \"kubernetes.io/projected/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-kube-api-access-lq7m9\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.610724 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.610791 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-trusted-ca\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.610791 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wvp\" (UniqueName: \"kubernetes.io/projected/70ab9d5e-f98a-4b5a-b72b-175e9182f488-kube-api-access-g9wvp\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.610791 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610776    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.610915 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzvz\" (UniqueName: \"kubernetes.io/projected/7184fff6-b600-467a-bd3b-f11fd6f6558f-kube-api-access-7tzvz\") pod \"network-check-source-6f78d59f99-fxq8l\" (UID: \"7184fff6-b600-467a-bd3b-f11fd6f6558f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"
Nov 03 11:50:39.610915 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmlvw\" (UniqueName: \"kubernetes.io/projected/8e156d6b-a0b5-460d-8946-7f7329753e54-kube-api-access-gmlvw\") pod \"kas-connection-checker-685895c7c9-xv5pl\" (UID: \"8e156d6b-a0b5-460d-8946-7f7329753e54\") " pod="kube-system/kas-connection-checker-685895c7c9-xv5pl"
Nov 03 11:50:39.611015 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.611015 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.610972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.611015 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.611135 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.611135 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611024    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c1607bc-d5e4-46b1-be5e-e27e7462a967-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.611135 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.611135 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.611333 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxks\" (UniqueName: \"kubernetes.io/projected/4f115611-61eb-45bb-9e13-34b7a2519dd5-kube-api-access-bxxks\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.611333 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gzp\" (UniqueName: \"kubernetes.io/projected/ef83543a-8e80-486b-adeb-4d5aad83d4b0-kube-api-access-z4gzp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.611333 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.611333 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncm5f\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.611333 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxzb\" (UniqueName: \"kubernetes.io/projected/4e9fac69-3de2-419e-931f-f1cb74a8aebb-kube-api-access-4lxzb\") pod \"volume-data-source-validator-58c9f5f46d-ldg6p\" (UID: \"4e9fac69-3de2-419e-931f-f1cb74a8aebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"
Nov 03 11:50:39.611333 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkfhq\" (UniqueName: \"kubernetes.io/projected/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-kube-api-access-gkfhq\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef83543a-8e80-486b-adeb-4d5aad83d4b0-serving-cert\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8522b9-5150-4e60-95bb-f9b04a71add9-config-volume\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-default-certificate\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70ab9d5e-f98a-4b5a-b72b-175e9182f488-serving-cert\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.611595 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8522b9-5150-4e60-95bb-f9b04a71add9-tmp-dir\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.611842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.611842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.611842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-stats-auth\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.611842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bab06cb5-f005-4c43-a247-5f38134286cb-config\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.611842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz5v\" (UniqueName: \"kubernetes.io/projected/bab06cb5-f005-4c43-a247-5f38134286cb-kube-api-access-qwz5v\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.611842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nf6r\" (UniqueName: \"kubernetes.io/projected/8c1607bc-d5e4-46b1-be5e-e27e7462a967-kube-api-access-7nf6r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.611842 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bab06cb5-f005-4c43-a247-5f38134286cb-serving-cert\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-service-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-tmp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.611989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-snapshots\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c1607bc-d5e4-46b1-be5e-e27e7462a967-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzzd\" (UniqueName: \"kubernetes.io/projected/358ed64f-afa7-41b0-9e8f-d981bebf2750-kube-api-access-5wzzd\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.612164 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkkf\" (UniqueName: \"kubernetes.io/projected/7b7eef80-4dd6-4087-b74c-66adfb323841-kube-api-access-xqkkf\") pod \"kas-connection-checker-685895c7c9-n7zd8\" (UID: \"7b7eef80-4dd6-4087-b74c-66adfb323841\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zd8"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612250    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bab06cb5-f005-4c43-a247-5f38134286cb-config\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612363    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-tmp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdkb\" (UniqueName: \"kubernetes.io/projected/4a8522b9-5150-4e60-95bb-f9b04a71add9-kube-api-access-czdkb\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hsb\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612450    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-service-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.612459    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.612472    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.612513    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.112501297 +0000 UTC m=+31.632995401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612561    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.612603 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfk8\" (UniqueName: \"kubernetes.io/projected/9e19744a-acf7-4e34-957f-c01efa20f0b2-kube-api-access-wkfk8\") pod \"kas-connection-checker-685895c7c9-nct5z\" (UID: \"9e19744a-acf7-4e34-957f-c01efa20f0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-nct5z"
Nov 03 11:50:39.613223 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.612617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358ed64f-afa7-41b0-9e8f-d981bebf2750-cert\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:39.613223 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.613180    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-snapshots\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.615723 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.615686    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.615769 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.615688    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.636914 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.636890    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 11:50:39.637008 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.636967    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.637606    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.637635    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.637769    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.637827    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638179    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638252    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638356    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638438    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638463    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638602    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzgk2\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638725    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638749    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638887    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tvsxn\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.638890    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.639081    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-whvfc\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.639143    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nxjhp\""
Nov 03 11:50:39.640942 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.639677    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 11:50:39.641879 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.641859    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.642062 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.642042    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.642644 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.642629    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:39.642733 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.642688    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.14266961 +0000 UTC m=+31.663163717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:39.643971 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.643954    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef83543a-8e80-486b-adeb-4d5aad83d4b0-serving-cert\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.644061 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.644045    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bab06cb5-f005-4c43-a247-5f38134286cb-serving-cert\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.644232 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.644215    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c1607bc-d5e4-46b1-be5e-e27e7462a967-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.668438 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.668419    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gzp\" (UniqueName: \"kubernetes.io/projected/ef83543a-8e80-486b-adeb-4d5aad83d4b0-kube-api-access-z4gzp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.668544 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.668515    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmlvw\" (UniqueName: \"kubernetes.io/projected/8e156d6b-a0b5-460d-8946-7f7329753e54-kube-api-access-gmlvw\") pod \"kas-connection-checker-685895c7c9-xv5pl\" (UID: \"8e156d6b-a0b5-460d-8946-7f7329753e54\") " pod="kube-system/kas-connection-checker-685895c7c9-xv5pl"
Nov 03 11:50:39.671387 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.671364    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.671683 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.671658    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxks\" (UniqueName: \"kubernetes.io/projected/4f115611-61eb-45bb-9e13-34b7a2519dd5-kube-api-access-bxxks\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:39.673233 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.673190    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nf6r\" (UniqueName: \"kubernetes.io/projected/8c1607bc-d5e4-46b1-be5e-e27e7462a967-kube-api-access-7nf6r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.673657 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.673632    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfk8\" (UniqueName: \"kubernetes.io/projected/9e19744a-acf7-4e34-957f-c01efa20f0b2-kube-api-access-wkfk8\") pod \"kas-connection-checker-685895c7c9-nct5z\" (UID: \"9e19744a-acf7-4e34-957f-c01efa20f0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-nct5z"
Nov 03 11:50:39.673756 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.673632    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hsb\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:39.674082 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.674062    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkkf\" (UniqueName: \"kubernetes.io/projected/7b7eef80-4dd6-4087-b74c-66adfb323841-kube-api-access-xqkkf\") pod \"kas-connection-checker-685895c7c9-n7zd8\" (UID: \"7b7eef80-4dd6-4087-b74c-66adfb323841\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zd8"
Nov 03 11:50:39.674132 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.674066    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz5v\" (UniqueName: \"kubernetes.io/projected/bab06cb5-f005-4c43-a247-5f38134286cb-kube-api-access-qwz5v\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.713535 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncm5f\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.713616 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxzb\" (UniqueName: \"kubernetes.io/projected/4e9fac69-3de2-419e-931f-f1cb74a8aebb-kube-api-access-4lxzb\") pod \"volume-data-source-validator-58c9f5f46d-ldg6p\" (UID: \"4e9fac69-3de2-419e-931f-f1cb74a8aebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"
Nov 03 11:50:39.713616 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkfhq\" (UniqueName: \"kubernetes.io/projected/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-kube-api-access-gkfhq\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.713616 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8522b9-5150-4e60-95bb-f9b04a71add9-config-volume\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.713677    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-default-certificate\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70ab9d5e-f98a-4b5a-b72b-175e9182f488-serving-cert\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.713747    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.213734605 +0000 UTC m=+31.734228712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:39.713779 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8522b9-5150-4e60-95bb-f9b04a71add9-tmp-dir\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.713789    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.713846    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.213829725 +0000 UTC m=+31.734323833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.713872    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.213864835 +0000 UTC m=+31.734358955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-stats-auth\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzzd\" (UniqueName: \"kubernetes.io/projected/358ed64f-afa7-41b0-9e8f-d981bebf2750-kube-api-access-5wzzd\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.713986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdkb\" (UniqueName: \"kubernetes.io/projected/4a8522b9-5150-4e60-95bb-f9b04a71add9-kube-api-access-czdkb\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714072    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8522b9-5150-4e60-95bb-f9b04a71add9-tmp-dir\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.714198 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358ed64f-afa7-41b0-9e8f-d981bebf2750-cert\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be887ffe-4891-4231-93ee-13d03c4efd85-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-config\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714264    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8522b9-5150-4e60-95bb-f9b04a71add9-config-volume\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7m9\" (UniqueName: \"kubernetes.io/projected/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-kube-api-access-lq7m9\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.714452    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.714496    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.214480103 +0000 UTC m=+31.734974214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.714689    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.714758    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.214743816 +0000 UTC m=+31.735237931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714786    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714880    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.714894    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:39.714903 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.714909    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:39.715489 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:39.714950    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:40.214935058 +0000 UTC m=+31.735429169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:39.715489 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715168    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be887ffe-4891-4231-93ee-13d03c4efd85-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:39.715489 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715254    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.715489 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.714367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-trusted-ca\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.715489 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wvp\" (UniqueName: \"kubernetes.io/projected/70ab9d5e-f98a-4b5a-b72b-175e9182f488-kube-api-access-g9wvp\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.715489 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715374    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-config\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.715489 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzvz\" (UniqueName: \"kubernetes.io/projected/7184fff6-b600-467a-bd3b-f11fd6f6558f-kube-api-access-7tzvz\") pod \"network-check-source-6f78d59f99-fxq8l\" (UID: \"7184fff6-b600-467a-bd3b-f11fd6f6558f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"
Nov 03 11:50:39.715902 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.715902 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.715902 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715692    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-trusted-ca\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.715902 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.715670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.716275 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.716247    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.716578 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.716558    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.717018 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.716998    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70ab9d5e-f98a-4b5a-b72b-175e9182f488-serving-cert\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.717093 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.717025    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358ed64f-afa7-41b0-9e8f-d981bebf2750-cert\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:39.717093 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.717049    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-default-certificate\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.718091 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.718066    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-stats-auth\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.718341 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.718326    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.751418 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.751397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nct5z"
Nov 03 11:50:39.758567 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.758552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-n7zd8"
Nov 03 11:50:39.767625 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.767602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xv5pl"
Nov 03 11:50:39.773482 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.773429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncm5f\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.775162 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.775063    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdkb\" (UniqueName: \"kubernetes.io/projected/4a8522b9-5150-4e60-95bb-f9b04a71add9-kube-api-access-czdkb\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:39.776000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.775540    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkfhq\" (UniqueName: \"kubernetes.io/projected/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-kube-api-access-gkfhq\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:39.776000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.775582    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wvp\" (UniqueName: \"kubernetes.io/projected/70ab9d5e-f98a-4b5a-b72b-175e9182f488-kube-api-access-g9wvp\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.776000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.775594    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7m9\" (UniqueName: \"kubernetes.io/projected/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-kube-api-access-lq7m9\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:39.776000 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.775960    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:39.776278 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.776177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzzd\" (UniqueName: \"kubernetes.io/projected/358ed64f-afa7-41b0-9e8f-d981bebf2750-kube-api-access-5wzzd\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:39.776375 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.776353    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxzb\" (UniqueName: \"kubernetes.io/projected/4e9fac69-3de2-419e-931f-f1cb74a8aebb-kube-api-access-4lxzb\") pod \"volume-data-source-validator-58c9f5f46d-ldg6p\" (UID: \"4e9fac69-3de2-419e-931f-f1cb74a8aebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"
Nov 03 11:50:39.777067 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.777034    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzvz\" (UniqueName: \"kubernetes.io/projected/7184fff6-b600-467a-bd3b-f11fd6f6558f-kube-api-access-7tzvz\") pod \"network-check-source-6f78d59f99-fxq8l\" (UID: \"7184fff6-b600-467a-bd3b-f11fd6f6558f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"
Nov 03 11:50:39.777725 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.777688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:39.799511 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.799482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:39.809865 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.809639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:39.828097 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.828038    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:39.852713 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.846106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"
Nov 03 11:50:39.852713 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.852499    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"
Nov 03 11:50:39.861843 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:39.861658    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:40.137042 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.137015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:40.137431 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.137163    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:40.137431 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.137180    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:40.137431 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.137235    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.137216991 +0000 UTC m=+32.657711083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:40.138077 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.138056    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7eef80_4dd6_4087_b74c_66adfb323841.slice/crio-ab0f12e617bf9c3f08341101f631dd6d7041844d61f4d55075eacf6add68b707 WatchSource:0}: Error finding container ab0f12e617bf9c3f08341101f631dd6d7041844d61f4d55075eacf6add68b707: Status 404 returned error can't find the container with id ab0f12e617bf9c3f08341101f631dd6d7041844d61f4d55075eacf6add68b707
Nov 03 11:50:40.138380 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.138362    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n7zd8"]
Nov 03 11:50:40.147571 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.147543    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e19744a_acf7_4e34_957f_c01efa20f0b2.slice/crio-b1a8ebdb08f5a60d42e2b2abf4e9ee8ffb3323eead6bbedd9edf2fc19c1bcb58 WatchSource:0}: Error finding container b1a8ebdb08f5a60d42e2b2abf4e9ee8ffb3323eead6bbedd9edf2fc19c1bcb58: Status 404 returned error can't find the container with id b1a8ebdb08f5a60d42e2b2abf4e9ee8ffb3323eead6bbedd9edf2fc19c1bcb58
Nov 03 11:50:40.147817 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.147794    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nct5z"]
Nov 03 11:50:40.150205 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.150003    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e156d6b_a0b5_460d_8946_7f7329753e54.slice/crio-d2e5e8ae872c493759b2a4c80d9e7a66f46e1c30f34edd3ed497bd64badc71d0 WatchSource:0}: Error finding container d2e5e8ae872c493759b2a4c80d9e7a66f46e1c30f34edd3ed497bd64badc71d0: Status 404 returned error can't find the container with id d2e5e8ae872c493759b2a4c80d9e7a66f46e1c30f34edd3ed497bd64badc71d0
Nov 03 11:50:40.150272 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.150220    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xv5pl"]
Nov 03 11:50:40.186029 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.185923    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xv5pl" event={"ID":"8e156d6b-a0b5-460d-8946-7f7329753e54","Type":"ContainerStarted","Data":"d2e5e8ae872c493759b2a4c80d9e7a66f46e1c30f34edd3ed497bd64badc71d0"}
Nov 03 11:50:40.188201 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.188125    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nct5z" event={"ID":"9e19744a-acf7-4e34-957f-c01efa20f0b2","Type":"ContainerStarted","Data":"b1a8ebdb08f5a60d42e2b2abf4e9ee8ffb3323eead6bbedd9edf2fc19c1bcb58"}
Nov 03 11:50:40.190484 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.190417    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-n7zd8" event={"ID":"7b7eef80-4dd6-4087-b74c-66adfb323841","Type":"ContainerStarted","Data":"ab0f12e617bf9c3f08341101f631dd6d7041844d61f4d55075eacf6add68b707"}
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.237642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.237759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.237791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.237836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.237882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.237919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.237949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238098    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238154    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.2381381 +0000 UTC m=+32.758632211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238221    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.238244537 +0000 UTC m=+32.758738643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238309    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238339    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.238329353 +0000 UTC m=+32.758823458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238404    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238433    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.238424261 +0000 UTC m=+32.758918369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:40.238577 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238521    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:40.239528 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.238546705 +0000 UTC m=+32.759040811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:40.239528 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.238565037 +0000 UTC m=+32.759059137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:40.239528 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238648    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:40.239528 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238662    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:40.239528 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:40.238720    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:41.238685569 +0000 UTC m=+32.759179663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:40.253874 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.253851    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"]
Nov 03 11:50:40.256568 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.256550    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"]
Nov 03 11:50:40.264610 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.264588    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab06cb5_f005_4c43_a247_5f38134286cb.slice/crio-39cf94d4c1f4b3757aa32580c75f4c610d1b86c3defb4eb12bd680aa3c039984 WatchSource:0}: Error finding container 39cf94d4c1f4b3757aa32580c75f4c610d1b86c3defb4eb12bd680aa3c039984: Status 404 returned error can't find the container with id 39cf94d4c1f4b3757aa32580c75f4c610d1b86c3defb4eb12bd680aa3c039984
Nov 03 11:50:40.265955 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.265733    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1607bc_d5e4_46b1_be5e_e27e7462a967.slice/crio-a643f286e34c55cca754636121422574aaa01cc712b2c812d0608ec0d6b50800 WatchSource:0}: Error finding container a643f286e34c55cca754636121422574aaa01cc712b2c812d0608ec0d6b50800: Status 404 returned error can't find the container with id a643f286e34c55cca754636121422574aaa01cc712b2c812d0608ec0d6b50800
Nov 03 11:50:40.292179 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.292155    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358ed64f_afa7_41b0_9e8f_d981bebf2750.slice/crio-74b06e47e3edc1c52ea2a0f3459e2c3e96df67823f4fdf1e771f24c5fd018fcf WatchSource:0}: Error finding container 74b06e47e3edc1c52ea2a0f3459e2c3e96df67823f4fdf1e771f24c5fd018fcf: Status 404 returned error can't find the container with id 74b06e47e3edc1c52ea2a0f3459e2c3e96df67823f4fdf1e771f24c5fd018fcf
Nov 03 11:50:40.292899 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.292878    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7j2qr"]
Nov 03 11:50:40.299750 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.299723    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7184fff6_b600_467a_bd3b_f11fd6f6558f.slice/crio-53ca057e6a76e750f1f26c50a4954e9aa873fbb5c9037bdc81e3369369347760 WatchSource:0}: Error finding container 53ca057e6a76e750f1f26c50a4954e9aa873fbb5c9037bdc81e3369369347760: Status 404 returned error can't find the container with id 53ca057e6a76e750f1f26c50a4954e9aa873fbb5c9037bdc81e3369369347760
Nov 03 11:50:40.301718 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.301681    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"]
Nov 03 11:50:40.306046 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.306023    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"]
Nov 03 11:50:40.306908 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.306885    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ab9d5e_f98a_4b5a_b72b_175e9182f488.slice/crio-c8f1bb76fd3f8dc327bb1fa86cdc7659e091716c8d7367d960d4780d4f1a0571 WatchSource:0}: Error finding container c8f1bb76fd3f8dc327bb1fa86cdc7659e091716c8d7367d960d4780d4f1a0571: Status 404 returned error can't find the container with id c8f1bb76fd3f8dc327bb1fa86cdc7659e091716c8d7367d960d4780d4f1a0571
Nov 03 11:50:40.325853 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.325833    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9fac69_3de2_419e_931f_f1cb74a8aebb.slice/crio-08753b452fd13ca07767e4901ab1ea782166159e2bb1905793917a47c4dac1ca WatchSource:0}: Error finding container 08753b452fd13ca07767e4901ab1ea782166159e2bb1905793917a47c4dac1ca: Status 404 returned error can't find the container with id 08753b452fd13ca07767e4901ab1ea782166159e2bb1905793917a47c4dac1ca
Nov 03 11:50:40.327802 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.327780    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"]
Nov 03 11:50:40.360517 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:40.360496    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef83543a_8e80_486b_adeb_4d5aad83d4b0.slice/crio-354a80d8561f3174771fb28dc8ac3bb1ff4425ba3cb74fe6a7cdc774988652fc WatchSource:0}: Error finding container 354a80d8561f3174771fb28dc8ac3bb1ff4425ba3cb74fe6a7cdc774988652fc: Status 404 returned error can't find the container with id 354a80d8561f3174771fb28dc8ac3bb1ff4425ba3cb74fe6a7cdc774988652fc
Nov 03 11:50:40.365400 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:40.365372    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8cwrc"]
Nov 03 11:50:41.002419 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.002388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:41.002577 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.002497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:41.002658 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.002644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:41.057602 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.057578    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 11:50:41.057750 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.057729    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xf5fg\""
Nov 03 11:50:41.057841 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.057763    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm8xm\""
Nov 03 11:50:41.057841 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.057788    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 11:50:41.058028 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.058011    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2ks4w\""
Nov 03 11:50:41.145684 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.145658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:41.146015 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.145804    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:41.146015 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.145819    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:41.146015 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.145882    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.145867234 +0000 UTC m=+34.666361328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:41.193835 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.193806    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-n7zd8" event={"ID":"7b7eef80-4dd6-4087-b74c-66adfb323841","Type":"ContainerStarted","Data":"eb82a8592674dd4448d5da2d50cf1136336703dc7856ef93d752add676ba42bf"}
Nov 03 11:50:41.194888 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.194865    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t" event={"ID":"bab06cb5-f005-4c43-a247-5f38134286cb","Type":"ContainerStarted","Data":"39cf94d4c1f4b3757aa32580c75f4c610d1b86c3defb4eb12bd680aa3c039984"}
Nov 03 11:50:41.196189 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.196162    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xv5pl" event={"ID":"8e156d6b-a0b5-460d-8946-7f7329753e54","Type":"ContainerStarted","Data":"fc5e3bdf10f193af2d4f1ca46c588bc8b2ff0db898c112181925c3efefb3ab46"}
Nov 03 11:50:41.197262 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.197243    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p" event={"ID":"4e9fac69-3de2-419e-931f-f1cb74a8aebb","Type":"ContainerStarted","Data":"08753b452fd13ca07767e4901ab1ea782166159e2bb1905793917a47c4dac1ca"}
Nov 03 11:50:41.198464 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.198445    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nct5z" event={"ID":"9e19744a-acf7-4e34-957f-c01efa20f0b2","Type":"ContainerStarted","Data":"956566921d6d574301e962c61a7b13664b1b966e311b535c0b7a5050d870acdc"}
Nov 03 11:50:41.199585 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.199564    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7j2qr" event={"ID":"358ed64f-afa7-41b0-9e8f-d981bebf2750","Type":"ContainerStarted","Data":"74b06e47e3edc1c52ea2a0f3459e2c3e96df67823f4fdf1e771f24c5fd018fcf"}
Nov 03 11:50:41.200683 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.200666    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l" event={"ID":"7184fff6-b600-467a-bd3b-f11fd6f6558f","Type":"ContainerStarted","Data":"53ca057e6a76e750f1f26c50a4954e9aa873fbb5c9037bdc81e3369369347760"}
Nov 03 11:50:41.201805 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.201782    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" event={"ID":"70ab9d5e-f98a-4b5a-b72b-175e9182f488","Type":"ContainerStarted","Data":"c8f1bb76fd3f8dc327bb1fa86cdc7659e091716c8d7367d960d4780d4f1a0571"}
Nov 03 11:50:41.202841 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.202815    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m" event={"ID":"8c1607bc-d5e4-46b1-be5e-e27e7462a967","Type":"ContainerStarted","Data":"a643f286e34c55cca754636121422574aaa01cc712b2c812d0608ec0d6b50800"}
Nov 03 11:50:41.203771 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.203753    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8cwrc" event={"ID":"ef83543a-8e80-486b-adeb-4d5aad83d4b0","Type":"ContainerStarted","Data":"354a80d8561f3174771fb28dc8ac3bb1ff4425ba3cb74fe6a7cdc774988652fc"}
Nov 03 11:50:41.246953 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.246929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:41.247071 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.246974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:41.247071 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.247004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:41.247071 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.247035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247093    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.247073639 +0000 UTC m=+34.767567750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247131    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.247143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247163    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247186    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.247155066 +0000 UTC m=+34.767649162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247096    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247204    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:41.247226 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247213    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:41.247602 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247215    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.247199208 +0000 UTC m=+34.767693314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:41.247602 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247265    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.247252951 +0000 UTC m=+34.767747048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:41.247602 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247282    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.24727298 +0000 UTC m=+34.767767079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:41.247602 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.247346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:41.247602 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.247369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:41.247863 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247728    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:41.247863 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247758    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.247749893 +0000 UTC m=+34.768243984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:41.247863 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247800    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:41.247863 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.247823    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:43.247817053 +0000 UTC m=+34.768311145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:41.576664 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.576619    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xv5pl" podStartSLOduration=443.576601756 podStartE2EDuration="7m23.576601756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:41.575198411 +0000 UTC m=+33.095692526" watchObservedRunningTime="2026-11-03 11:50:41.576601756 +0000 UTC m=+33.097095871"
Nov 03 11:50:41.695716 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.689333    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-n7zd8" podStartSLOduration=443.689313486 podStartE2EDuration="7m23.689313486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:41.686889463 +0000 UTC m=+33.207383575" watchObservedRunningTime="2026-11-03 11:50:41.689313486 +0000 UTC m=+33.209807601"
Nov 03 11:50:41.954858 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.954733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:41.954858 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.954799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:41.955077 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.954941    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:41.955077 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:41.955001    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:13.954983512 +0000 UTC m=+65.475477604 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : secret "metrics-daemon-secret" not found
Nov 03 11:50:41.965849 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:41.965797    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:42.055499 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:42.055304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:42.059624 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:42.059598    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:42.241520 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:42.241447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:42.261096 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:42.261074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:42.685391 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:42.685287    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nct5z" podStartSLOduration=444.685264746 podStartE2EDuration="7m24.685264746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:41.804507158 +0000 UTC m=+33.325001274" watchObservedRunningTime="2026-11-03 11:50:42.685264746 +0000 UTC m=+34.205758862"
Nov 03 11:50:42.686482 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:42.686455    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkt2k"]
Nov 03 11:50:42.725309 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:42.725285    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsvz7"]
Nov 03 11:50:42.729523 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:42.729496    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098926dd_2788_41f4_a964_5f7448c8c41f.slice/crio-a74fb912ce09deffd951613efab05c6fe8c8c4c048278c764b28e4a7e25dbcfb WatchSource:0}: Error finding container a74fb912ce09deffd951613efab05c6fe8c8c4c048278c764b28e4a7e25dbcfb: Status 404 returned error can't find the container with id a74fb912ce09deffd951613efab05c6fe8c8c4c048278c764b28e4a7e25dbcfb
Nov 03 11:50:43.167926 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.167889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:43.168105 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.168041    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:43.168105 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.168061    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:43.168228 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.168124    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.168107598 +0000 UTC m=+38.688601692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:43.214453 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.214417    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkt2k" event={"ID":"ad5bba67-4cce-447f-aa79-d6061b83fb63","Type":"ContainerStarted","Data":"96cff9a58ccf86b2514b3046877db97d70deba98b8aacf78d7b3659fa4fe5ac5"}
Nov 03 11:50:43.216195 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.216145    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsvz7" event={"ID":"098926dd-2788-41f4-a964-5f7448c8c41f","Type":"ContainerStarted","Data":"a74fb912ce09deffd951613efab05c6fe8c8c4c048278c764b28e4a7e25dbcfb"}
Nov 03 11:50:43.268682 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.268652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:43.269129 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.268901    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:43.269129 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.268965    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.268945789 +0000 UTC m=+38.789439897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.268726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.269363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.269408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.269448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.269489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:43.269554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269717    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269765    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.269751261 +0000 UTC m=+38.790245367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269782    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.269772867 +0000 UTC m=+38.790266960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269827    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269841    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269866    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.269855475 +0000 UTC m=+38.790349577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269882    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.269872981 +0000 UTC m=+38.790367082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269930    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:43.270072 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269990    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.269950426 +0000 UTC m=+38.790444526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:43.271068 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.269996    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:43.271068 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.270011    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:43.271068 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:43.270055    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.2700421 +0000 UTC m=+38.790536208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:44.224411 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:44.224373    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p" event={"ID":"4e9fac69-3de2-419e-931f-f1cb74a8aebb","Type":"ContainerStarted","Data":"4642705aa32092e4290ffefe3d00a0218be1d54e3f9fbcd619ad0db4abcf2fd6"}
Nov 03 11:50:44.339343 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:44.339279    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p" podStartSLOduration=403.29374829 podStartE2EDuration="6m46.339266122s" totalImagesPullingTime="3.045517832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:44.338436994 +0000 UTC m=+35.858931108" watchObservedRunningTime="2026-11-03 11:50:44.339266122 +0000 UTC m=+35.859760259"
Nov 03 11:50:45.233901 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:45.232476    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/0.log"
Nov 03 11:50:45.233901 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:45.232522    2581 generic.go:356] "Generic (PLEG): container finished" podID="70ab9d5e-f98a-4b5a-b72b-175e9182f488" containerID="73af758c27142ec8aff03d6059e06d865cb2b6d50353227127e17c81b6454fea" exitCode=255
Nov 03 11:50:45.233901 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:45.233294    2581 scope.go:122] "RemoveContainer" containerID="73af758c27142ec8aff03d6059e06d865cb2b6d50353227127e17c81b6454fea"
Nov 03 11:50:45.233901 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:45.233568    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" event={"ID":"70ab9d5e-f98a-4b5a-b72b-175e9182f488","Type":"ContainerDied","Data":"73af758c27142ec8aff03d6059e06d865cb2b6d50353227127e17c81b6454fea"}
Nov 03 11:50:46.239261 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:46.239231    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/1.log"
Nov 03 11:50:46.239882 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:46.239853    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/0.log"
Nov 03 11:50:46.240002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:46.239899    2581 generic.go:356] "Generic (PLEG): container finished" podID="70ab9d5e-f98a-4b5a-b72b-175e9182f488" containerID="16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454" exitCode=255
Nov 03 11:50:46.240002 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:46.239949    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" event={"ID":"70ab9d5e-f98a-4b5a-b72b-175e9182f488","Type":"ContainerDied","Data":"16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454"}
Nov 03 11:50:46.240117 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:46.240037    2581 scope.go:122] "RemoveContainer" containerID="73af758c27142ec8aff03d6059e06d865cb2b6d50353227127e17c81b6454fea"
Nov 03 11:50:46.240244 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:46.240224    2581 scope.go:122] "RemoveContainer" containerID="16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454"
Nov 03 11:50:46.240481 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:46.240456    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rmmzr_openshift-console-operator(70ab9d5e-f98a-4b5a-b72b-175e9182f488)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" podUID="70ab9d5e-f98a-4b5a-b72b-175e9182f488"
Nov 03 11:50:47.208479 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.208351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:47.208638 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.208587    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:47.208638 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.208613    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:47.208779 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.208681    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.208661425 +0000 UTC m=+46.729155541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:47.255267 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.255190    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7j2qr" event={"ID":"358ed64f-afa7-41b0-9e8f-d981bebf2750","Type":"ContainerStarted","Data":"5f55f6fe731a3aaf85ba1e765d127a24e65099f30a4e3784cd619c85d76abfb7"}
Nov 03 11:50:47.257219 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.257195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/1.log"
Nov 03 11:50:47.257512 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.257495    2581 scope.go:122] "RemoveContainer" containerID="16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454"
Nov 03 11:50:47.257719 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.257677    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rmmzr_openshift-console-operator(70ab9d5e-f98a-4b5a-b72b-175e9182f488)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" podUID="70ab9d5e-f98a-4b5a-b72b-175e9182f488"
Nov 03 11:50:47.259299 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.259262    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m" event={"ID":"8c1607bc-d5e4-46b1-be5e-e27e7462a967","Type":"ContainerStarted","Data":"cbb9c1644f40746f2056286aac789c89044f86e5257b41f051b4fac467511d2a"}
Nov 03 11:50:47.260976 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.260957    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8cwrc" event={"ID":"ef83543a-8e80-486b-adeb-4d5aad83d4b0","Type":"ContainerStarted","Data":"59902d0ab67d6106e17e92018899165a7e695e415605d4352af1604975de2c2e"}
Nov 03 11:50:47.262450 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.262392    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t" event={"ID":"bab06cb5-f005-4c43-a247-5f38134286cb","Type":"ContainerStarted","Data":"e0ac9de693deae2341e2c3fe7e854ecc2d191944f775e48fc7b0359cdc7076f5"}
Nov 03 11:50:47.308863 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.308838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:47.308956 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.308880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:47.308956 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.308922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:47.309040 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.308955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:47.309040 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.308965    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:47.309040 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.308988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:47.309040 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.309019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309044    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309060    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.309042121 +0000 UTC m=+46.829536235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309068    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309096    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.309081327 +0000 UTC m=+46.829575422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309105    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309118    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.309109299 +0000 UTC m=+46.829603398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309141    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.309132695 +0000 UTC m=+46.829626791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309142    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309153    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:47.309169 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309167    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.309159495 +0000 UTC m=+46.829653604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:47.309538 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.309190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:47.309538 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309240    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:47.309538 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309245    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.309236996 +0000 UTC m=+46.829731114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:47.309538 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:47.309274    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.309258656 +0000 UTC m=+46.829752756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:47.369663 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.369611    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7j2qr" podStartSLOduration=1.692357463 podStartE2EDuration="8.369593255s" totalImagesPullingTime="6.677235792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:47.368644162 +0000 UTC m=+38.889138276" watchObservedRunningTime="2026-11-03 11:50:47.369593255 +0000 UTC m=+38.890087370"
Nov 03 11:50:47.485048 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.484954    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-8cwrc" podStartSLOduration=398.893463067 podStartE2EDuration="6m45.484934921s" totalImagesPullingTime="6.591471854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:47.484083458 +0000 UTC m=+39.004577573" watchObservedRunningTime="2026-11-03 11:50:47.484934921 +0000 UTC m=+39.005429034"
Nov 03 11:50:47.598899 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:47.598847    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t" podStartSLOduration=396.911464652 podStartE2EDuration="6m43.598829965s" totalImagesPullingTime="6.687365313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:47.5973195 +0000 UTC m=+39.117813615" watchObservedRunningTime="2026-11-03 11:50:47.598829965 +0000 UTC m=+39.119324082"
Nov 03 11:50:48.267949 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.267913    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkt2k" event={"ID":"ad5bba67-4cce-447f-aa79-d6061b83fb63","Type":"ContainerStarted","Data":"10bb90e648867c012a74e1db9a08e7d8f42e00049ac3cf4720c30e7d28117eda"}
Nov 03 11:50:48.268478 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.268048    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:48.269927 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.269895    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l" event={"ID":"7184fff6-b600-467a-bd3b-f11fd6f6558f","Type":"ContainerStarted","Data":"bb8f866088c5cbc2d9dd9b07be6464ee47fb0de8dfb3a3d62236e8106fa3b736"}
Nov 03 11:50:48.384066 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.384017    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gkt2k" podStartSLOduration=34.592255 podStartE2EDuration="39.384003185s" totalImagesPullingTime="4.791748185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:48.383543906 +0000 UTC m=+39.904038019" watchObservedRunningTime="2026-11-03 11:50:48.384003185 +0000 UTC m=+39.904497299"
Nov 03 11:50:48.384644 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.384620    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m" podStartSLOduration=398.698261802 podStartE2EDuration="6m45.384611982s" totalImagesPullingTime="6.68635018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:47.825946095 +0000 UTC m=+39.346440210" watchObservedRunningTime="2026-11-03 11:50:48.384611982 +0000 UTC m=+39.905106095"
Nov 03 11:50:48.499218 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.499159    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l" podStartSLOduration=403.46967539 podStartE2EDuration="6m50.499142366s" totalImagesPullingTime="7.029466976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:48.498132957 +0000 UTC m=+40.018627071" watchObservedRunningTime="2026-11-03 11:50:48.499142366 +0000 UTC m=+40.019636479"
Nov 03 11:50:48.518636 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.518536    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"]
Nov 03 11:50:48.547975 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.547944    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bq6v7"]
Nov 03 11:50:48.548143 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.548101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"
Nov 03 11:50:48.562050 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.562028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.574118 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.574093    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"]
Nov 03 11:50:48.607878 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.607852    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bq6v7"]
Nov 03 11:50:48.624635 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.624606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hjg\" (UniqueName: \"kubernetes.io/projected/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3-kube-api-access-z7hjg\") pod \"migrator-66df5d496b-g4hph\" (UID: \"85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"
Nov 03 11:50:48.663680 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.663650    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 11:50:48.663858 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.663750    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fw7f8\""
Nov 03 11:50:48.663858 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.663844    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:48.663994 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.663851    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 11:50:48.663994 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.663896    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 11:50:48.663994 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.663915    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7npm\""
Nov 03 11:50:48.726070 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.726037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxx4d\" (UniqueName: \"kubernetes.io/projected/eccf3810-774d-4fe5-9516-6c08440c1505-kube-api-access-jxx4d\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.726212 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.726141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hjg\" (UniqueName: \"kubernetes.io/projected/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3-kube-api-access-z7hjg\") pod \"migrator-66df5d496b-g4hph\" (UID: \"85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"
Nov 03 11:50:48.726280 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.726241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eccf3810-774d-4fe5-9516-6c08440c1505-crio-socket\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.726400 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.726374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eccf3810-774d-4fe5-9516-6c08440c1505-data-volume\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.726537 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.726427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.726537 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.726520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.784064 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.783996    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hjg\" (UniqueName: \"kubernetes.io/projected/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3-kube-api-access-z7hjg\") pod \"migrator-66df5d496b-g4hph\" (UID: \"85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"
Nov 03 11:50:48.827643 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.827612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eccf3810-774d-4fe5-9516-6c08440c1505-crio-socket\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.827794 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.827745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eccf3810-774d-4fe5-9516-6c08440c1505-data-volume\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.827794 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.827758    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eccf3810-774d-4fe5-9516-6c08440c1505-crio-socket\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.827794 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.827791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.827952 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.827824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.827952 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.827865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxx4d\" (UniqueName: \"kubernetes.io/projected/eccf3810-774d-4fe5-9516-6c08440c1505-kube-api-access-jxx4d\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.827952 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:48.827947    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 11:50:48.828096 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.828027    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eccf3810-774d-4fe5-9516-6c08440c1505-data-volume\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:48.828458 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:48.828410    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:48.828458 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:48.828444    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:49.328018526 +0000 UTC m=+40.848512634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : configmap "kube-rbac-proxy" not found
Nov 03 11:50:48.828640 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:48.828490    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:49.328471753 +0000 UTC m=+40.848965869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:48.871024 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.871001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"
Nov 03 11:50:48.886174 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:48.886151    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxx4d\" (UniqueName: \"kubernetes.io/projected/eccf3810-774d-4fe5-9516-6c08440c1505-kube-api-access-jxx4d\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:49.261557 ip-10-0-131-136 kubenswrapper[2581]: W1103 11:50:49.261522    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85af2d99_dfe2_4630_b5f7_cf96ca0ca5b3.slice/crio-bf37c77e8dabef686900480a3f554f696398086e7687c2fe04d370bb623e8b75 WatchSource:0}: Error finding container bf37c77e8dabef686900480a3f554f696398086e7687c2fe04d370bb623e8b75: Status 404 returned error can't find the container with id bf37c77e8dabef686900480a3f554f696398086e7687c2fe04d370bb623e8b75
Nov 03 11:50:49.262457 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:49.262387    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"]
Nov 03 11:50:49.274508 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:49.274476    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph" event={"ID":"85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3","Type":"ContainerStarted","Data":"bf37c77e8dabef686900480a3f554f696398086e7687c2fe04d370bb623e8b75"}
Nov 03 11:50:49.333525 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:49.333129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:49.333525 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:49.333193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:49.333525 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:49.333369    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:49.333525 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:49.333423    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.333406854 +0000 UTC m=+41.853900967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:49.333525 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:49.333456    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 11:50:49.333525 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:49.333501    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.333489156 +0000 UTC m=+41.853983264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : configmap "kube-rbac-proxy" not found
Nov 03 11:50:49.829236 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:49.829204    2581 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:49.829390 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:49.829272    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:49.829643 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:49.829627    2581 scope.go:122] "RemoveContainer" containerID="16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454"
Nov 03 11:50:49.829857 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:49.829837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rmmzr_openshift-console-operator(70ab9d5e-f98a-4b5a-b72b-175e9182f488)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" podUID="70ab9d5e-f98a-4b5a-b72b-175e9182f488"
Nov 03 11:50:50.343907 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:50.343710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:50.343907 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:50.343787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:50.343907 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:50.343834    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 11:50:50.343907 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:50.343912    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.343891009 +0000 UTC m=+43.864385100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : configmap "kube-rbac-proxy" not found
Nov 03 11:50:50.344388 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:50.343922    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:50.344388 ip-10-0-131-136 kubenswrapper[2581]: E1103 11:50:50.343976    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.343961975 +0000 UTC m=+43.864456067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:50.923553 ip-10-0-131-136 kubenswrapper[2581]: I1103 11:50:50.923441    2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:50.923652 ip-10-0-131-136 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 11:50:50.936147 ip-10-0-131-136 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 11:50:50.936316 ip-10-0-131-136 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 11:50:50.936367 ip-10-0-131-136 systemd[1]: kubelet.service: Consumed 2.896s CPU time.
Nov 03 11:50:50.949397 ip-10-0-131-136 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 11:50:51.029828 ip-10-0-131-136 kubenswrapper[4909]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:51.029828 ip-10-0-131-136 kubenswrapper[4909]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 11:50:51.029828 ip-10-0-131-136 kubenswrapper[4909]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:51.029828 ip-10-0-131-136 kubenswrapper[4909]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:51.031846 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031768    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.031846 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031844    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.031846 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031848    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031852    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031855    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031860    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031863    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031866    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031869    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031872    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031874    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031878    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031880    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031883    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031886    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031889    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031891    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031894    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031896    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031899    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031902    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.031951 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031904    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031907    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031910    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031913    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031915    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031921    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031924    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031927    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031929    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031932    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031935    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031938    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031940    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031943    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031946    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031948    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031951    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031953    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031956    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031959    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.032417 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031962    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031964    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031967    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031970    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031972    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031975    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031978    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031980    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031983    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031986    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031988    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031991    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031993    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031996    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.031999    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032001    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032004    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032007    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032010    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032012    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.032922 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032015    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032018    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032022    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032025    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032028    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032031    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032033    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032036    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032039    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032042    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032044    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032047    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032050    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032053    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032055    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032058    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032060    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032063    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032065    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032068    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.033396 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032071    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032073    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032076    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032078    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032081    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032083    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032086    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032088    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032091    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032094    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032097    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032100    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032102    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032105    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032108    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032112    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032114    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032117    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032120    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.033898 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032122    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032126    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032130    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032133    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032138    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032141    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032953    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032962    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032965    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032969    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032971    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032974    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032977    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032980    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032982    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032985    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032988    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032991    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032993    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.034369 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032996    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.032999    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033002    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033005    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033008    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033016    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033018    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033021    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033025    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033027    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033030    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033032    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033035    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033038    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033040    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033043    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033045    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033048    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033050    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.034863 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033053    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033056    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033058    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033060    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033063    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033065    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033068    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033070    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033073    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033075    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033078    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033081    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033083    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033086    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033089    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033092    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033097    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033101    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033103    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.035334 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033107    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033110    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033113    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033116    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033118    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033121    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033124    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033127    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033130    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033132    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033135    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033137    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033140    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033144    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033148    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033151    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033154    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033157    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033159    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033162    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.035809 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033165    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033167    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033170    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033173    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033176    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033179    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033181    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033184    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033187    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033189    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033192    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033194    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033197    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033200    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033203    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033206    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033209    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033211    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033214    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.036354 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033216    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033218    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033221    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033224    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033227    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033229    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033232    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033234    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033237    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033239    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033242    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033245    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033247    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033250    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033253    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033255    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033740    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033746    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033749    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033752    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.036832 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033755    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033757    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033760    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033763    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033765    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033768    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033771    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033777    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033780    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033782    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033785    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033790    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033793    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033796    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033798    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033801    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033804    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033807    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033809    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.037316 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033812    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033815    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033817    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033820    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033822    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033825    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033828    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033830    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033833    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033836    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033839    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033841    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033844    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033846    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033849    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033851    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033854    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033856    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033859    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033861    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.037787 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033866    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033869    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033871    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033874    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033878    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033880    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033883    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033885    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033888    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033891    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033893    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033896    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033899    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033902    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033905    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033908    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033910    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033913    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033916    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.038297 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033919    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033921    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033924    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033926    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033928    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033931    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033934    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033936    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033941    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033945    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033947    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033951    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033954    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033957    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033960    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033963    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033966    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033970    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033972    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.038766 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033975    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033978    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033980    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033983    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033986    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033989    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033991    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033994    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033997    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.033999    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034002    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034006    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034009    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034012    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034015    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034017    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034020    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034022    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034025    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.039232 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034027    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034031    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034033    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034036    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034038    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034041    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034135    4909 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034144    4909 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034150    4909 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034156    4909 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034160    4909 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034165    4909 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034169    4909 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034173    4909 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034177    4909 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034180    4909 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034183    4909 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034186    4909 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034189    4909 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034193    4909 flags.go:64] FLAG: --cgroup-root=""
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034195    4909 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034199    4909 flags.go:64] FLAG: --client-ca-file=""
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034201    4909 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 11:50:51.039734 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034205    4909 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034209    4909 flags.go:64] FLAG: --cluster-domain=""
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034212    4909 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034215    4909 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034218    4909 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034221    4909 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034226    4909 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034229    4909 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034232    4909 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034250    4909 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034255    4909 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034258    4909 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034262    4909 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034265    4909 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034268    4909 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034273    4909 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034278    4909 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034281    4909 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034284    4909 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034290    4909 flags.go:64] FLAG: --enable-server="true"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034295    4909 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034300    4909 flags.go:64] FLAG: --event-burst="100"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034303    4909 flags.go:64] FLAG: --event-qps="50"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034306    4909 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 11:50:51.040301 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034309    4909 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034313    4909 flags.go:64] FLAG: --eviction-hard=""
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034317    4909 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034320    4909 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034323    4909 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034326    4909 flags.go:64] FLAG: --eviction-soft=""
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034329    4909 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034332    4909 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034335    4909 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034338    4909 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034342    4909 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034344    4909 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034347    4909 flags.go:64] FLAG: --feature-gates=""
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034351    4909 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034354    4909 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034358    4909 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034361    4909 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034364    4909 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034367    4909 flags.go:64] FLAG: --help="false"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034370    4909 flags.go:64] FLAG: --hostname-override="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034373    4909 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034377    4909 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034380    4909 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 11:50:51.041050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034383    4909 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034388    4909 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034391    4909 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034394    4909 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034397    4909 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034402    4909 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034405    4909 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034408    4909 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034411    4909 flags.go:64] FLAG: --kube-reserved=""
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034415    4909 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034417    4909 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034420    4909 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034423    4909 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034426    4909 flags.go:64] FLAG: --lock-file=""
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034429    4909 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034432    4909 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034435    4909 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034441    4909 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034444    4909 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034447    4909 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034450    4909 flags.go:64] FLAG: --logging-format="text"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034453    4909 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034456    4909 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034459    4909 flags.go:64] FLAG: --manifest-url=""
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034462    4909 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 11:50:51.042044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034467    4909 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034470    4909 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034473    4909 flags.go:64] FLAG: --max-pods="110"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034476    4909 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034480    4909 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034483    4909 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034486    4909 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034489    4909 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034493    4909 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034496    4909 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034503    4909 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034507    4909 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034511    4909 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034515    4909 flags.go:64] FLAG: --pod-cidr=""
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034518    4909 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034521    4909 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034524    4909 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034527    4909 flags.go:64] FLAG: --port="10250"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034530    4909 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034533    4909 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-02c4211575cb30411"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034536    4909 flags.go:64] FLAG: --qos-reserved=""
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034539    4909 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034542    4909 flags.go:64] FLAG: --register-node="true"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034546    4909 flags.go:64] FLAG: --register-with-taints=""
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034549    4909 flags.go:64] FLAG: --registry-burst="10"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034552    4909 flags.go:64] FLAG: --registry-qps="5"
Nov 03 11:50:51.042753 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034555    4909 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034557    4909 flags.go:64] FLAG: --reserved-memory=""
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034561    4909 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034564    4909 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034568    4909 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034571    4909 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034573    4909 flags.go:64] FLAG: --runonce="false"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034576    4909 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034579    4909 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034582    4909 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034585    4909 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034588    4909 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034592    4909 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034595    4909 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034599    4909 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034602    4909 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034605    4909 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034608    4909 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034613    4909 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034616    4909 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034619    4909 flags.go:64] FLAG: --system-cgroups=""
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034622    4909 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034628    4909 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034631    4909 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034634    4909 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 11:50:51.043452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034637    4909 flags.go:64] FLAG: --tls-min-version=""
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034640    4909 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034643    4909 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034646    4909 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034649    4909 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034652    4909 flags.go:64] FLAG: --v="2"
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034656    4909 flags.go:64] FLAG: --version="false"
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034660    4909 flags.go:64] FLAG: --vmodule=""
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034664    4909 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.034667    4909 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034828    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034832    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034835    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034838    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034840    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034843    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034846    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034849    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034851    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034854    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034857    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.044158 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034862    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034867    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034870    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034874    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034878    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034880    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034883    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034887    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034889    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034893    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034897    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034900    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034903    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034906    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034909    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034912    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034915    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034918    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034921    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.044833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034924    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034926    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034929    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034931    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034934    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034937    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034939    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034942    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034944    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034947    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034949    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034952    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034955    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034958    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034961    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034963    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034966    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034970    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034972    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034975    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.045359 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034978    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034981    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034984    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034986    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034989    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034991    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034994    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.034997    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035000    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035002    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035005    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035008    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035010    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035013    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035015    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035018    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035020    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035023    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035026    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.046129 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035029    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035031    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035034    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035036    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035039    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035041    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035044    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035046    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035049    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035052    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035055    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035058    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035060    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035063    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035066    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035069    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035071    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035074    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035077    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035079    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.047000 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035082    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035085    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035087    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035090    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035092    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035095    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035097    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035100    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035102    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035105    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035108    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035110    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035113    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035115    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035118    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035121    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.047636 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035123    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.035149    4909 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035188    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035193    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035195    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035198    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035201    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035205    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035208    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035211    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035213    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035216    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035219    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035222    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035224    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035227    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.048151 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035230    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035232    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035235    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035238    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035240    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035243    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035246    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035249    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035251    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035254    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035257    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035259    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035262    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035265    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035268    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035270    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035273    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035276    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035278    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035281    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.048595 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035283    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035286    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035288    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035292    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035295    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035298    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035300    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035303    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035305    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035308    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035311    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035313    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035316    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035319    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035322    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035324    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035327    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035330    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035333    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035336    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.049088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035338    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035341    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035351    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035354    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035357    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035361    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035363    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035366    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035369    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035371    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035374    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035376    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035379    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035381    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035384    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035387    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035390    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035392    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035395    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.049583 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035398    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035400    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035403    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035406    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035409    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035413    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035416    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035419    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035422    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035425    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035428    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035430    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035433    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035435    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035439    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035443    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035446    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035448    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035452    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.050064 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035455    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035457    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035460    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035463    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035467    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035469    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035472    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035474    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035477    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035480    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035482    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035485    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035487    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.035490    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.039102    4909 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.039116    4909 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.039134    4909 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.039139    4909 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039193    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.050573 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039198    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039203    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039207    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039210    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039213    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039216    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039219    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039221    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039224    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039227    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039229    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039232    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039235    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039238    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039241    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039244    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039247    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039250    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039252    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039255    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.051096 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039258    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039260    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039263    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039266    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039268    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039271    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039274    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039277    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039280    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039282    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039285    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039288    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039290    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039293    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039296    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039299    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039301    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039304    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039306    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.051592 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039309    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039311    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039314    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039316    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039319    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039321    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039324    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039326    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039329    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039332    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039335    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039337    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039340    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039349    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039353    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039355    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039358    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039360    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039363    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.052075 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039366    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039368    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039371    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039374    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039376    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039379    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039382    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039384    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039387    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039390    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039392    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039395    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039398    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039400    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039403    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039405    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039408    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039410    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039413    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039415    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.052529 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039418    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039420    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039423    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039426    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039429    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039431    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039434    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039436    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039439    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039441    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039444    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039447    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039449    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039452    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039456    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039460    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039463    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039466    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039469    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039472    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.053023 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039474    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039477    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039479    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039482    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039485    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039487    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039490    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.039517    4909 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039661    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039666    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039669    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039672    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039675    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039678    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039680    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.053499 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039683    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039686    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039688    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039692    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039707    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039711    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039714    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039717    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039719    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039722    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039725    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039727    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039730    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039732    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039735    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039738    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039741    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039744    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039747    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.053880 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039750    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039752    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039755    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039757    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039760    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039765    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039767    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039770    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039773    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039776    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039778    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039781    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039784    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039786    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039789    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039791    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039794    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039797    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039799    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039802    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.054357 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039804    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039808    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039811    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039814    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039817    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039820    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039824    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039828    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039832    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039836    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039841    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039845    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039848    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039851    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039853    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039855    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039858    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039863    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039866    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039869    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.054852 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039871    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039874    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039876    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039879    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039883    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039886    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039888    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039891    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039894    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039896    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039899    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039901    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039904    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039907    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039909    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039912    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039914    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039917    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039920    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039922    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.055335 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039924    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039927    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039930    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039933    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039936    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039938    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039941    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039944    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039946    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039951    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039954    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039956    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039959    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039962    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039964    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039967    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039971    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039974    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039977    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.056022 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.039979    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.040003    4909 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040035    4909 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040039    4909 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040043    4909 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040046    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040048    4909 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040051    4909 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040053    4909 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040056    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040059    4909 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040061    4909 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040064    4909 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040066    4909 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040069    4909 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040072    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:51.056483 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040074    4909 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040077    4909 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040080    4909 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040083    4909 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040086    4909 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040088    4909 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040091    4909 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040095    4909 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040097    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040100    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040103    4909 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040105    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040108    4909 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040112    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040114    4909 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040117    4909 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040119    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040122    4909 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040125    4909 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040128    4909 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:51.057018 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040130    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040133    4909 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040135    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040138    4909 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040140    4909 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040143    4909 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040146    4909 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040148    4909 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040151    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040153    4909 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040156    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040159    4909 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040161    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040164    4909 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040167    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040169    4909 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040172    4909 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040175    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040178    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040181    4909 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:51.057886 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040184    4909 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040187    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040189    4909 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040192    4909 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040195    4909 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040199    4909 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040201    4909 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040204    4909 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040206    4909 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040209    4909 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040212    4909 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040215    4909 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040218    4909 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040220    4909 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040224    4909 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040229    4909 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040232    4909 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040235    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040238    4909 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:51.058492 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040240    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040243    4909 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040246    4909 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040248    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040251    4909 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040253    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040256    4909 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040258    4909 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040262    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040265    4909 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040267    4909 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040270    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040274    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040276    4909 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040279    4909 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040282    4909 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040284    4909 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040287    4909 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040291    4909 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040294    4909 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:51.058966 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040296    4909 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040299    4909 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040302    4909 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040304    4909 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040307    4909 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040310    4909 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040313    4909 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040315    4909 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040318    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040321    4909 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040323    4909 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040326    4909 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:50:51.040330    4909 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.040378    4909 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.042264    4909 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.042323    4909 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.042506    4909 server.go:1014] "Starting client certificate rotation"
Nov 03 11:50:51.059555 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.042642    4909 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.042684    4909 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 11:45:08 +0000 UTC" deadline="2028-05-06 21:29:37.866993421 +0000 UTC"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.042732    4909 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13209h38m46.824263961s"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.042943    4909 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.043997    4909 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.045152    4909 log.go:25] "Validated CRI v1 runtime API"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.047163    4909 log.go:25] "Validated CRI v1 image API"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.047923    4909 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 11:50:51.060129 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.049649    4909 fs.go:135] Filesystem UUIDs: map[43b3e96e-0519-47e4-803f-06f2b5b72e14:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cffe2e66-271f-42fb-90a6-bb71261a2220:/dev/nvme0n1p3]
Nov 03 11:50:51.060505 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.049667    4909 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08753b452fd13ca07767e4901ab1ea782166159e2bb1905793917a47c4dac1ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08753b452fd13ca07767e4901ab1ea782166159e2bb1905793917a47c4dac1ca/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/254ca63a2723b5c0fe1ccae51c7c0f13eed703349d08bc8ab2a278059788e93b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/254ca63a2723b5c0fe1ccae51c7c0f13eed703349d08bc8ab2a278059788e93b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/281628f75b067a433c06ff16bdd0c7a98ab29c79d922f092875c0a049878eef3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/281628f75b067a433c06ff16bdd0c7a98ab29c79d922f092875c0a049878eef3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bb466f00cd21ebce72be2d5610ab182cb6b11efe172907f3d66976dd5664c99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bb466f00cd21ebce72be2d5610ab182cb6b11efe172907f3d66976dd5664c99/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/354a80d8561f3174771fb28dc8ac3bb1ff4425ba3cb74fe6a7cdc774988652fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/354a80d8561f3174771fb28dc8ac3bb1ff4425ba3cb74fe6a7cdc774988652fc/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39cf94d4c1f4b3757aa32580c75f4c610d1b86c3defb4eb12bd680aa3c039984/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39cf94d4c1f4b3757aa32580c75f4c610d1b86c3defb4eb12bd680aa3c039984/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53ca057e6a76e750f1f26c50a4954e9aa873fbb5c9037bdc81e3369369347760/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53ca057e6a76e750f1f26c50a4954e9aa873fbb5c9037bdc81e3369369347760/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61bdc96b360e4d56ebadaade76a91acc2452433bd30d184dd48e27ddfcc00593/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61bdc96b360e4d56ebadaade76a91acc2452433bd30d184dd48e27ddfcc00593/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63d24c65779b165b896ea304f0f73add40f26a0462206503242dd5944c25a7a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63d24c65779b165b896ea304f0f73add40f26a0462206503242dd5944c25a7a2/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74b06e47e3edc1c52ea2a0f3459e2c3e96df67823f4fdf1e771f24c5fd018fcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74b06e47e3edc1c52ea2a0f3459e2c3e96df67823f4fdf1e771f24c5fd018fcf/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dd2599074cd01041c6bec5f9e8a8d53b577f2c5d24cf29588e54c9ef9a13c9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dd2599074cd01041c6bec5f9e8a8d53b577f2c5d24cf29588e54c9ef9a13c9d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9351f15d426761591aede7a3eea7c78302a5453e1130a2a906aa95359236d0e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9351f15d426761591aede7a3eea7c78302a5453e1130a2a906aa95359236d0e3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96cff9a58ccf86b2514b3046877db97d70deba98b8aacf78d7b3659fa4fe5ac5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96cff9a58ccf86b2514b3046877db97d70deba98b8aacf78d7b3659fa4fe5ac5/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a643f286e34c55cca754636121422574aaa01cc712b2c812d0608ec0d6b50800/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a643f286e34c55cca754636121422574aaa01cc712b2c812d0608ec0d6b50800/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a74fb912ce09deffd951613efab05c6fe8c8c4c048278c764b28e4a7e25dbcfb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a74fb912ce09deffd951613efab05c6fe8c8c4c048278c764b28e4a7e25dbcfb/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab0f12e617bf9c3f08341101f631dd6d7041844d61f4d55075eacf6add68b707/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab0f12e617bf9c3f08341101f631dd6d7041844d61f4d55075eacf6add68b707/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adb334b78165ad565da8592a1ac6ce4e420f2cd162cdd1350924cd26e1f645a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adb334b78165ad565da8592a1ac6ce4e420f2cd162cdd1350924cd26e1f645a4/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1a8ebdb08f5a60d42e2b2abf4e9ee8ffb3323eead6bbedd9edf2fc19c1bcb58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1a8ebdb08f5a60d42e2b2abf4e9ee8ffb3323eead6bbedd9edf2fc19c1bcb58/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf37c77e8dabef686900480a3f554f696398086e7687c2fe04d370bb623e8b75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf37c77e8dabef686900480a3f554f696398086e7687c2fe04d370bb623e8b75/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8f1bb76fd3f8dc327bb1fa86cdc7659e091716c8d7367d960d4780d4f1a0571/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8f1bb76fd3f8dc327bb1fa86cdc7659e091716c8d7367d960d4780d4f1a0571/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2e5e8ae872c493759b2a4c80d9e7a66f46e1c30f34edd3ed497bd64badc71d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2e5e8ae872c493759b2a4c80d9e7a66f46e1c30f34edd3ed497bd64badc71d0/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/098926dd-2788-41f4-a964-5f7448c8c41f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/098926dd-2788-41f4-a964-5f7448c8c41f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~projected/kube-api-access-ncm5f:{mountpoint:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~projected/kube-api-access-ncm5f major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~projected/kube-api-access-g8hsb:{mountpoint:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~projected/kube-api-access-g8hsb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/358ed64f-afa7-41b0-9e8f-d981bebf2750/volumes/kubernetes.io~projected/kube-api-access-5wzzd:{mountpoint:/var/lib/kubelet/pods/358ed64f-afa7-41b0-9e8f-d981bebf2750/volumes/kubernetes.io~projected/kube-api-access-5wzzd major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/358ed64f-afa7-41b0-9e8f-d981bebf2750/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/358ed64f-afa7-41b0-9e8f-d981bebf2750/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a8522b9-5150-4e60-95bb-f9b04a71add9/volumes/kubernetes.io~projected/kube-api-access-czdkb:{mountpoint:/var/lib/kubelet/pods/4a8522b9-5150-4e60-95bb-f9b04a71add9/volumes/kubernetes.io~projected/kube-api-access-czdkb major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e9fac69-3de2-419e-931f-f1cb74a8aebb/volumes/kubernetes.io~projected/kube-api-access-4lxzb:{mountpoint:/var/lib/kubelet/pods/4e9fac69-3de2-419e-931f-f1cb74a8aebb/volumes/kubernetes.io~projected/kube-api-access-4lxzb major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f115611-61eb-45bb-9e13-34b7a2519dd5/volumes/kubernetes.io~projected/kube-api-access-bxxks:{mountpoint:/var/lib/kubelet/pods/4f115611-61eb-45bb-9e13-34b7a2519dd5/volumes/kubernetes.io~projected/kube-api-access-bxxks major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6/volumes/kubernetes.io~projected/kube-api-access-j9t6r:{mountpoint:/var/lib/kubelet/pods/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6/volumes/kubernetes.io~projected/kube-api-access-j9t6r major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70ab9d5e-f98a-4b5a-b72b-175e9182f488/volumes/kubernetes.io~projected/kube-api-access-g9wvp:{mountpoint:/var/lib/kubelet/pods/70ab9d5e-f98a-4b5a-b72b-175e9182f488/volumes/kubernetes.io~projected/kube-api-access-g9wvp major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70ab9d5e-f98a-4b5a-b72b-175e9182f488/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/70ab9d5e-f98a-4b5a-b72b-175e9182f488/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7184fff6-b600-467a-bd3b-f11fd6f6558f/volumes/kubernetes.io~projected/kube-api-access-7tzvz:{mountpoint:/var/lib/kubelet/pods/7184fff6-b600-467a-bd3b-f11fd6f6558f/volumes/kubernetes.io~projected/kube-api-access-7tzvz major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~projected/kube-api-access-gkfhq:{mountpoint:/var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~projected/kube-api-access-gkfhq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~secret/stats-auth major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77d106d7-59a7-459c-b012-aaa087cb50d9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/77d106d7-59a7-459c-b012-aaa087cb50d9/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b7eef80-4dd6-4087-b74c-66adfb323841/volumes/kubernetes.io~projected/kube-api-access-xqkkf:{mountpoint:/var/lib/kubelet/pods/7b7eef80-4dd6-4087-b74c-66adfb323841/volumes/kubernetes.io~projected/kube-api-access-xqkkf major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3/volumes/kubernetes.io~projected/kube-api-access-z7hjg:{mountpoint:/var/lib/kubelet/pods/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3/volumes/kubernetes.io~projected/kube-api-access-z7hjg major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volumes/kubernetes.io~projected/kube-api-access-ccdzg:{mountpoint:/var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volumes/kubernetes.io~projected/kube-api-access-ccdzg major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c1607bc-d5e4-46b1-be5e-e27e7462a967/volumes/kubernetes.io~projected/kube-api-access-7nf6r:{mountpoint:/var/lib/kubelet/pods/8c1607bc-d5e4-46b1-be5e-e27e7462a967/volumes/kubernetes.io~projected/kube-api-access-7nf6r major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c1607bc-d5e4-46b1-be5e-e27e7462a967/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8c1607bc-d5e4-46b1-be5e-e27e7462a967/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e156d6b-a0b5-460d-8946-7f7329753e54/volumes/kubernetes.io~projected/kube-api-access-gmlvw:{mountpoint:/var/lib/kubelet/pods/8e156d6b-a0b5-460d-8946-7f7329753e54/volumes/kubernetes.io~projected/kube-api-access-gmlvw major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ceda7ca-9770-42ec-973a-0435ff74dd29/volumes/kubernetes.io~projected/kube-api-access-lktl5:{mountpoint:/var/lib/kubelet/pods/9ceda7ca-9770-42ec-973a-0435ff74dd29/volumes/kubernetes.io~projected/kube-api-access-lktl5 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e19744a-acf7-4e34-957f-c01efa20f0b2/volumes/kubernetes.io~projected/kube-api-access-wkfk8:{mountpoint:/var/lib/kubelet/pods/9e19744a-acf7-4e34-957f-c01efa20f0b2/volumes/kubernetes.io~projected/kube-api-access-wkfk8 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a208df1b-dba2-4b61-bd72-d7a4dffcada0/volumes/kubernetes.io~projected/kube-api-access-5mn4m:{mountpoint:/var/lib/kubelet/pods/a208df1b-dba2-4b61-bd72-d7a4dffcada0/volumes/kubernetes.io~projected/kube-api-access-5mn4m major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad5bba67-4cce-447f-aa79-d6061b83fb63/volumes/kubernetes.io~projected/kube-api-access-g9pcf:{mountpoint:/var/lib/kubelet/pods/ad5bba67-4cce-447f-aa79-d6061b83fb63/volumes/kubernetes.io~projected/kube-api-access-g9pcf major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9319607-a6c8-4003-89bd-14da12ba17a8/volumes/kubernetes.io~projected/kube-api-access-hzxwp:{mountpoint:/var/lib/kubelet/pods/b9319607-a6c8-4003-89bd-14da12ba17a8/volumes/kubernetes.io~projected/kube-api-access-hzxwp major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bab06cb5-f005-4c43-a247-5f38134286cb/volumes/kubernetes.io~projected/kube-api-access-qwz5v:{mountpoint:/var/lib/kubelet/pods/bab06cb5-f005-4c43-a247-5f38134286cb/volumes/kubernetes.io~projected/kube-api-access-qwz5v major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bab06cb5-f005-4c43-a247-5f38134286cb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bab06cb5-f005-4c43-a247-5f38134286cb/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~projected/kube-api-access-smlbr:{mountpoint:/var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~projected/kube-api-access-smlbr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2/volumes/kubernetes.io~projected/kube-api-access-xkqw5:{mountpoint:/var/lib/kubelet/pods/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2/volumes/kubernetes.io~projected/kube-api-access-xkqw5 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8/volumes/kubernetes.io~projected/kube-api-access-lq7m9:{mountpoint:/var/lib/kubelet/pods/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8/volumes/kubernetes.io~projected/kube-api-access-lq7m9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d378c41e-4f5f-4f08-b051-23694fee59d4/volumes/kubernetes.io~projected/kube-api-access-nnq6q:{mountpoint:/var/lib/kubelet/pods/d378c41e-4f5f-4f08-b051-23694fee59d4/volumes/kubernetes.io~projected/kube-api-access-nnq6q major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1/volumes/kubernetes.io~projected/kube-api-access-zxpx9:{mountpoint:/var/lib/kubelet/pods/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1/volumes/kubernetes.io~projected/kube-api-access-zxpx9 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eccf3810-774d-4fe5-9516-6c08440c1505/volumes/kubernetes.io~projected/kube-api-access-jxx4d:{mountpoint:/var/lib/kubelet/pods/eccf3810-774d-4fe5-9516-6c08440c1505/volumes/kubernetes.io~projected/kube-api-access-jxx4d major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef83543a-8e80-486b-adeb-4d5aad83d4b0/volumes/kubernetes.io~projected/kube-api-access-z4gzp:{mountpoint:/var/lib/kubelet/pods/ef83543a-8e80-486b-adeb-4d5aad83d4b0/volumes/kubernetes.io~projected/kube-api-access-z4gzp major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef83543a-8e80-486b-adeb-4d5aad83d4b0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ef83543a-8e80-486b-adeb-4d5aad83d4b0/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/825c65b4d69774c447146760ae886d9b8598c6f7f3af74c5d882c1cd4972813e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/05b2650b566f4a9c697cb1b1f662215ab8111bfa2c47da671b63b4ec1993fcc4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e51bc1544f38ff9829d16620b81ab7260188578871b14c0e2a9c43d69baad94a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1f3a7a114826d90376abb9df3b96209bf3bf62609b943a5772d96a3499c51fde/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/27eed74d5a929fe18f55f751d893c0180649ba05d077a73786c1c23faf99e739/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0e2ab179f529f3d6c8e7ef09c2b8e93e720e1e6a35b80405423ae9b15bd1fbd8/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d1cc6f0fcaa8c22c0a9f63590649421f3a063aa34aeea5770fbda14f54347e9c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e4e509a31a15942a18823b735bbf6b6c7f3afbfcf3a03977d1bbe50fb006ecd9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9e5a9fba6f121ea76ad2bd372ef4381427cdec93fb5b027f023308feef919f25/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/135ded3e8ac278fd3dd3ffabef1d79e3ffcfb8d692eb5350fd2403b8760a0f8c/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/5c481d71012a4b9cd7140a955b42cc066fb4875d3f5e9d42f31ca39a41eac88f/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/9450a69f93d30bd71739d402027156b71c932d3c5fa50b54cd89024dbdc21379/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4db23d41995dbf687221aa51a825eb4309d6cbd71de1196f31c39b8462682b92/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9be2e85422a2b5751917b3e98e44d74786fcd54ca46651cd7af7a09ae73122a0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dca58992d3c842badef9dfaeafc7089ee6b2dff3bcb841fbb205155a4208c99f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9282b8fcdc309cc3773e4510805e94aa3f0caf07c0c46e0b37b5de801df75e7d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f74142bd44ef7d89c3f32af0cb43128ac60e63a463f1ef07af467d95b90255d5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/638f73b1006b337d2b177abe60fba5f6e657f6644728b6b8cca73214aeacf65b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b9f1bbd700fcadd025c9b89dd0550b4f42167848906ee8feb797a2da0a83fb9a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/95c601d447851df8d4dd503641404636051e3cf3c9e0081374f1869f8c9eadbc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/042575b8a1c6122f7f9a12c73fd81e09188d08e2d50def7fe0da7915a1cc5200/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/aea400994dce3b62095c7e2e512b2bfcb573e4fd5eb9925cbc039459f6c7ade8/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/ab56531575213df56c2432461ae062189c99d690c2ffebb2420fb70ea747f654/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/3892c55309a80b670b96ecf64acac18bc342ab0068cc584ce647f6d7ea95095d/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/1cdb8823e9c0909a05717c96c13b8e8d22b33057d4889bd29a10e1e2142aed2e/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/6c962bd4ad7f3b49d7ceae4fc716ae48292fe299bf27535717c910774508b70b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/6fe7717af60e2a03aa6608dee2ce48516ae194aa83ad953540b2c2c363888684/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1a812990bb5d65e7728c975161b643260a21a8a61bab929b76bcdde3bfb57f36/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/3edae5f146574b905c72317ccd7becee5049d96fbf525d8c53c1adb94aaa0c40/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/ca9bc2d6033f06d10f960f457aa5faf11033619d61ac44810ddc094135900074/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/a2ed5561fd94dc323c6d02966c40ffb98cb964ebab4a159cd90b30d9c8777f8f/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/09a0457ba6259ed28a7c057f3e2f68efe9b4747cac5924e15857dd9e010d9331/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/740af97611362b94bd1bceda33407878f4a5590eec3454d752b0f9f6068cb9e2/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/2af2267267bc90d7357c69ea3e36cfda18ccf30013d49d06825d15e600d2bac0/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/87d38221074e5758969d16f1ad73d5eaed5a21c0954218b9f2a051a792b39ba5/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/d87869e3b83882b559e076f01570613aee0dc9e4af92981f3be47a537a603e1a/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/229ec1472dfc27587fc726c53001191b5e2355f002a155f64fda78f647a90b36/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/5d964339174eda1996cac8f4e841718249d3f1e97adce38b751cd118028f5211/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/020db37feb537e60a788e99f923ee35c4512b67a95a818f567d7b91a6720f477/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/364650566d5ca314bc3766a8ffec66fd2a163a9a6c9eead3002b90a3b055e91b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/8daef0258e8334f1120441eab31ff64af6f7dc445e2e9c5d4f09e5c120960f98/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/3628cacdc1fe0d7af2710b6ea89443b4bef0158debd7a55afda9aede3338e382/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/cb4382c9a807ddd660934d1eab3ef94e1a80c39ba090eb3c6b8e1a6be675f649/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/f5595557c62674be2cf9a72389b72835761fd2ecd67526e9f33f9dd356c1c25d/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/c17d4b6c62e04acbb363deb8a15fdce27f04691e2fdca90464d985a6c1522f53/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/cbbe2f028f86ab87de81384648292232200607b643f4976c0f02c992b37c4b78/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/f8ee405fbcabb9368b015f4457af1cbf36ef8ba181c53ef528a4792b2383d3ff/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/57c19f3967afcbd9a4fd686c969f48744a47fc9896f4419d74cc2fbf063cea0b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/732b1c27940aea97e6ce27930f6908bc3a56686513abadb337fe4c82273048ee/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/fc81c9b3feee597ba457322b993585c08d986349eec5c60c9cde942a5acfe8b0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/88620eef5b02506a9f13cd55457baccae157f8be94e5d3f2aa1b09813cd0f3db/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e5ec7343973b35d46cc416537fc1a690c3184ba5fa0a6532df980e90f92719e0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/5ca1416f6f9e85ded75d58445d5adc46fae0dd52164d9ad94e8e6fdc15250661/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/c9b8e9e1b2c798cfd3149b7d07656e4f39352ca5f9f9f5b8efc61c81632613aa/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/684c119d817355745f17a7880f71dbc7554ff440a2635c565c5a6e54a457f967/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.056886    4909 manager.go:217] Machine: {Timestamp:2026-11-03 11:50:51.056288081 +0000 UTC m=+0.068754741 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090867 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22541a0be8d026063352a07a9ddc80 SystemUUID:ec22541a-0be8-d026-0633-52a07a9ddc80 BootID:f664fcb4-09b1-4ab4-a892-26d870e046ba Filesystems:[{Device:/var/lib/kubelet/pods/bab06cb5-f005-4c43-a247-5f38134286cb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74b06e47e3edc1c52ea2a0f3459e2c3e96df67823f4fdf1e771f24c5fd018fcf/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61bdc96b360e4d56ebadaade76a91acc2452433bd30d184dd48e27ddfcc00593/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef83543a-8e80-486b-adeb-4d5aad83d4b0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53ca057e6a76e750f1f26c50a4954e9aa873fbb5c9037bdc81e3369369347760/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96cff9a58ccf86b2514b3046877db97d70deba98b8aacf78d7b3659fa4fe5ac5/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bb466f00cd21ebce72be2d5610ab182cb6b11efe172907f3d66976dd5664c99/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/354a80d8561f3174771fb28dc8ac3bb1ff4425ba3cb74fe6a7cdc774988652fc/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77d106d7-59a7-459c-b012-aaa087cb50d9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70ab9d5e-f98a-4b5a-b72b-175e9182f488/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f115611-61eb-45bb-9e13-34b7a2519dd5/volumes/kubernetes.io~projected/kube-api-access-bxxks DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~projected/kube-api-access-gkfhq DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/098926dd-2788-41f4-a964-5f7448c8c41f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab0f12e617bf9c3f08341101f631dd6d7041844d61f4d55075eacf6add68b707/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/358ed64f-afa7-41b0-9e8f-d981bebf2750/volumes/kubernetes.io~projected/kube-api-access-5wzzd DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad5bba67-4cce-447f-aa79-d6061b83fb63/volumes/kubernetes.io~projected/kube-api-access-g9pcf DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d378c41e-4f5f-4f08-b051-23694fee59d4/volumes/kubernetes.io~projected/kube-api-access-nnq6q DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2/volumes/kubernetes.io~projected/kube-api-access-xkqw5 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2e5e8ae872c493759b2a4c80d9e7a66f46e1c30f34edd3ed497bd64badc71d0/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39cf94d4c1f4b3757aa32580c75f4c610d1b86c3defb4eb12bd680aa3c039984/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9319607-a6c8-4003-89bd-14da12ba17a8/volumes/kubernetes.io~projected/kube-api-access-hzxwp DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/254ca63a2723b5c0fe1ccae51c7c0f13eed703349d08bc8ab2a278059788e93b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8/volumes/kubernetes.io~projected/kube-api-access-lq7m9 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bab06cb5-f005-4c43-a247-5f38134286cb/volumes/kubernetes.io~projected/kube-api-access-qwz5v DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8f1bb76fd3f8dc327bb1fa86cdc7659e091716c8d7367d960d4780d4f1a0571/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dd2599074cd01041c6bec5f9e8a8d53b577f2c5d24cf29588e54c9ef9a13c9d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6/volumes/kubernetes.io~projected/kube-api-access-j9t6r DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes/kubernetes.io~projected/kube-api-access-g8hsb DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~projected/kube-api-access-ncm5f DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a8522b9-5150-4e60-95bb-f9b04a71add9/volumes/kubernetes.io~projected/kube-api-access-czdkb DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b7eef80-4dd6-4087-b74c-66adfb323841/volumes/kubernetes.io~projected/kube-api-access-xqkkf DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e9fac69-3de2-419e-931f-f1cb74a8aebb/volumes/kubernetes.io~projected/kube-api-access-4lxzb DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a74fb912ce09deffd951613efab05c6fe8c8c4c048278c764b28e4a7e25dbcfb/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a208df1b-dba2-4b61-bd72-d7a4dffcada0/volumes/kubernetes.io~projected/kube-api-access-5mn4m DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9351f15d426761591aede7a3eea7c78302a5453e1130a2a906aa95359236d0e3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c1607bc-d5e4-46b1-be5e-e27e7462a967/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71b27f5e-a0e1-4ba7-a4af-8282a4639d41/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ceda7ca-9770-42ec-973a-0435ff74dd29/volumes/kubernetes.io~projected/kube-api-access-lktl5 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3/volumes/kubernetes.io~projected/kube-api-access-z7hjg DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eccf3810-774d-4fe5-9516-6c08440c1505/volumes/kubernetes.io~projected/kube-api-access-jxx4d DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf37c77e8dabef686900480a3f554f696398086e7687c2fe04d370bb623e8b75/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63d24c65779b165b896ea304f0f73add40f26a0462206503242dd5944c25a7a2/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef83543a-8e80-486b-adeb-4d5aad83d4b0/volumes/kubernetes.io~projected/kube-api-access-z4gzp DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e19744a-acf7-4e34-957f-c01efa20f0b2/volumes/kubernetes.io~projected/kube-api-access-wkfk8 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70ab9d5e-f98a-4b5a-b72b-175e9182f488/volumes/kubernetes.io~projected/kube-api-access-g9wvp DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volumes/kubernetes.io~projected/kube-api-access-ccdzg DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7184fff6-b600-467a-bd3b-f11fd6f6558f/volumes/kubernetes.io~projected/kube-api-access-7tzvz DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1a8ebdb08f5a60d42e2b2abf4e9ee8ffb3323eead6bbedd9edf2fc19c1bcb58/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/874b36e5-1db3-498f-a51c-e23ccfb5947b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8c1607bc-d5e4-46b1-be5e-e27e7462a967/volumes/kubernetes.io~projected/kube-api-access-7nf6r DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e156d6b-a0b5-460d-8946-7f7329753e54/volumes/kubernetes.io~projected/kube-api-access-gmlvw DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a643f286e34c55cca754636121422574aaa01cc712b2c812d0608ec0d6b50800/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08753b452fd13ca07767e4901ab1ea782166159e2bb1905793917a47c4dac1ca/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/358ed64f-afa7-41b0-9e8f-d981bebf2750/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1/volumes/kubernetes.io~projected/kube-api-access-zxpx9 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/281628f75b067a433c06ff16bdd0c7a98ab29c79d922f092875c0a049878eef3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adb334b78165ad565da8592a1ac6ce4e420f2cd162cdd1350924cd26e1f645a4/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfd61052-ba77-4b91-bb1b-9d808791b96b/volumes/kubernetes.io~projected/kube-api-access-smlbr DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08753b452fd13ca MacAddress:02:83:32:0d:69:7a Speed:10000 Mtu:8901} {Name:354a80d8561f317 MacAddress:06:c3:2a:4c:76:2c Speed:10000 Mtu:8901} {Name:39cf94d4c1f4b37 MacAddress:ba:85:fe:9d:e6:95 Speed:10000 Mtu:8901} {Name:53ca057e6a76e75 MacAddress:e6:88:3d:1d:94:18 Speed:10000 Mtu:8901} {Name:74b06e47e3edc1c MacAddress:2a:70:25:de:b1:38 Speed:10000 Mtu:8901} {Name:96cff9a58ccf86b MacAddress:5e:57:3a:8f:fc:e1 Speed:10000 Mtu:8901} {Name:a643f286e34c55c MacAddress:0e:7c:a2:2b:9c:cd Speed:10000 Mtu:8901} {Name:a74fb912ce09def MacAddress:3e:25:57:b9:da:cf Speed:10000 Mtu:8901} {Name:ab0f12e617bf9c3 MacAddress:d2:87:4a:c4:11:46 Speed:10000 Mtu:8901} {Name:b1a8ebdb08f5a60 MacAddress:b6:85:b8:e3:4a:93 Speed:10000 Mtu:8901} {Name:bf37c77e8dabef6 MacAddress:46:41:4b:ff:0a:80 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f9:a8:c1:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:56:54:ea:33:26:b4 Speed:0 Mtu:8901} {Name:c8f1bb76fd3f8dc MacAddress:92:f7:0c:8f:02:d5 Speed:10000 Mtu:8901} {Name:d2e5e8ae872c493 MacAddress:c2:34:99:85:49:ce Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f9:a8:c1:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a2:f5:fb:02:17:7a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:44:c7:1c:d4:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.057744    4909 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.057802    4909 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058363    4909 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058386    4909 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-136.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058507    4909 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058515    4909 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058528    4909 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058547    4909 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058567    4909 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058738    4909 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058839    4909 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058872    4909 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058887    4909 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058896    4909 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058905    4909 kubelet.go:405] "Adding apiserver pod source"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.058912    4909 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.059798    4909 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.059811    4909 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:51.062217 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.060269    4909 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 11:50:51.074749 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.074734    4909 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 11:50:51.074749 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.074746    4909 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:51.074860 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.074845    4909 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 11:50:51.075021 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075011    4909 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 11:50:51.075055 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075039    4909 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 11:50:51.075138 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075129    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 11:50:51.075176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075142    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 11:50:51.075176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075148    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 11:50:51.075176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075154    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 11:50:51.075176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075159    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 11:50:51.075176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075165    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 11:50:51.075176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075170    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 11:50:51.075176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075176    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 11:50:51.075454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075182    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 11:50:51.075454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075188    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 11:50:51.075454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075201    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 11:50:51.075454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075211    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 11:50:51.075454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075237    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 11:50:51.075454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.075243    4909 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 11:50:51.078523 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.078512    4909 server.go:1263] "Started kubelet"
Nov 03 11:50:51.078630 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.078605    4909 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 11:50:51.078708 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.078611    4909 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 11:50:51.078708 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.078690    4909 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 11:50:51.079602 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.078968    4909 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 11:50:51.079037 ip-10-0-131-136 systemd[1]: Started Kubernetes Kubelet.
Nov 03 11:50:51.080139 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.080122    4909 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.083605    4909 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.083643    4909 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.083753    4909 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:09 +0000 UTC" deadline="2028-07-22 13:14:21.72420213 +0000 UTC"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.083796    4909 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15049h23m30.640410204s"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.084546    4909 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.084561    4909 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.084712    4909 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 11:50:51.085527 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.084960    4909 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.us-west-2.compute.internal\" not found"
Nov 03 11:50:51.087201 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.087181    4909 factory.go:55] Registering systemd factory
Nov 03 11:50:51.087201 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.087204    4909 factory.go:223] Registration of the systemd container factory successfully
Nov 03 11:50:51.087637 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.087619    4909 factory.go:153] Registering CRI-O factory
Nov 03 11:50:51.087637 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.087639    4909 factory.go:223] Registration of the crio container factory successfully
Nov 03 11:50:51.087845 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.087828    4909 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 11:50:51.087934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.087861    4909 factory.go:103] Registering Raw factory
Nov 03 11:50:51.087934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.087895    4909 manager.go:1199] Started watching for new ooms in manager
Nov 03 11:50:51.088395 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088378    4909 manager.go:319] Starting recovery of all containers
Nov 03 11:50:51.088475 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088409    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef83543a-8e80-486b-adeb-4d5aad83d4b0" volumeName="kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 11:50:51.088475 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088449    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ceda7ca-9770-42ec-973a-0435ff74dd29" volumeName="kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-daemon-config" seLinuxMountContext=""
Nov 03 11:50:51.088475 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088470    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="098926dd-2788-41f4-a964-5f7448c8c41f" volumeName="kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret" seLinuxMountContext=""
Nov 03 11:50:51.088562 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088485    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8522b9-5150-4e60-95bb-f9b04a71add9" volumeName="kubernetes.io/empty-dir/4a8522b9-5150-4e60-95bb-f9b04a71add9-tmp-dir" seLinuxMountContext=""
Nov 03 11:50:51.088562 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088502    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c1607bc-d5e4-46b1-be5e-e27e7462a967" volumeName="kubernetes.io/configmap/8c1607bc-d5e4-46b1-be5e-e27e7462a967-config" seLinuxMountContext=""
Nov 03 11:50:51.088562 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088519    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d378c41e-4f5f-4f08-b051-23694fee59d4" volumeName="kubernetes.io/configmap/d378c41e-4f5f-4f08-b051-23694fee59d4-serviceca" seLinuxMountContext=""
Nov 03 11:50:51.088562 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088535    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eccf3810-774d-4fe5-9516-6c08440c1505" volumeName="kubernetes.io/empty-dir/eccf3810-774d-4fe5-9516-6c08440c1505-data-volume" seLinuxMountContext=""
Nov 03 11:50:51.088562 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088552    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f923698-d3d2-4118-8f54-eb36c5f29983" volumeName="kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca" seLinuxMountContext=""
Nov 03 11:50:51.088693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088567    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f923698-d3d2-4118-8f54-eb36c5f29983" volumeName="kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f" seLinuxMountContext=""
Nov 03 11:50:51.088693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088583    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188932f7-09f2-4a38-94ec-8db245a5c89a" volumeName="kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca" seLinuxMountContext=""
Nov 03 11:50:51.088693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088597    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c1607bc-d5e4-46b1-be5e-e27e7462a967" volumeName="kubernetes.io/projected/8c1607bc-d5e4-46b1-be5e-e27e7462a967-kube-api-access-7nf6r" seLinuxMountContext=""
Nov 03 11:50:51.088693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088615    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfd61052-ba77-4b91-bb1b-9d808791b96b" volumeName="kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-tuned" seLinuxMountContext=""
Nov 03 11:50:51.088693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088631    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2" volumeName="kubernetes.io/projected/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-kube-api-access-xkqw5" seLinuxMountContext=""
Nov 03 11:50:51.088693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088644    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef83543a-8e80-486b-adeb-4d5aad83d4b0" volumeName="kubernetes.io/secret/ef83543a-8e80-486b-adeb-4d5aad83d4b0-serving-cert" seLinuxMountContext=""
Nov 03 11:50:51.088693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088665    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188932f7-09f2-4a38-94ec-8db245a5c89a" volumeName="kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088710    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b7eef80-4dd6-4087-b74c-66adfb323841" volumeName="kubernetes.io/projected/7b7eef80-4dd6-4087-b74c-66adfb323841-kube-api-access-xqkkf" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088725    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9319607-a6c8-4003-89bd-14da12ba17a8" volumeName="kubernetes.io/projected/b9319607-a6c8-4003-89bd-14da12ba17a8-kube-api-access-hzxwp" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088744    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab06cb5-f005-4c43-a247-5f38134286cb" volumeName="kubernetes.io/secret/bab06cb5-f005-4c43-a247-5f38134286cb-serving-cert" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088762    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2" volumeName="kubernetes.io/empty-dir/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-tmp-dir" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088778    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eccf3810-774d-4fe5-9516-6c08440c1505" volumeName="kubernetes.io/projected/eccf3810-774d-4fe5-9516-6c08440c1505-kube-api-access-jxx4d" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088793    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8522b9-5150-4e60-95bb-f9b04a71add9" volumeName="kubernetes.io/configmap/4a8522b9-5150-4e60-95bb-f9b04a71add9-config-volume" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088806    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874b36e5-1db3-498f-a51c-e23ccfb5947b" volumeName="kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-config" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088822    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad5bba67-4cce-447f-aa79-d6061b83fb63" volumeName="kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088839    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef83543a-8e80-486b-adeb-4d5aad83d4b0" volumeName="kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-tmp" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088852    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab9d5e-f98a-4b5a-b72b-175e9182f488" volumeName="kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-config" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088868    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874b36e5-1db3-498f-a51c-e23ccfb5947b" volumeName="kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088884    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ceda7ca-9770-42ec-973a-0435ff74dd29" volumeName="kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-cni-binary-copy" seLinuxMountContext=""
Nov 03 11:50:51.088934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088898    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188932f7-09f2-4a38-94ec-8db245a5c89a" volumeName="kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088937    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e9fac69-3de2-419e-931f-f1cb74a8aebb" volumeName="kubernetes.io/projected/4e9fac69-3de2-419e-931f-f1cb74a8aebb-kube-api-access-4lxzb" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088955    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71b27f5e-a0e1-4ba7-a4af-8282a4639d41" volumeName="kubernetes.io/projected/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-kube-api-access-gkfhq" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.088972    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ceda7ca-9770-42ec-973a-0435ff74dd29" volumeName="kubernetes.io/projected/9ceda7ca-9770-42ec-973a-0435ff74dd29-kube-api-access-lktl5" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089013    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a208df1b-dba2-4b61-bd72-d7a4dffcada0" volumeName="kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089029    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef83543a-8e80-486b-adeb-4d5aad83d4b0" volumeName="kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-service-ca-bundle" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089045    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188932f7-09f2-4a38-94ec-8db245a5c89a" volumeName="kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089063    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188932f7-09f2-4a38-94ec-8db245a5c89a" volumeName="kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089083    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f115611-61eb-45bb-9e13-34b7a2519dd5" volumeName="kubernetes.io/configmap/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-config" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089098    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77d106d7-59a7-459c-b012-aaa087cb50d9" volumeName="kubernetes.io/configmap/77d106d7-59a7-459c-b012-aaa087cb50d9-konnectivity-ca" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089114    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a208df1b-dba2-4b61-bd72-d7a4dffcada0" volumeName="kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-binary-copy" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089132    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f923698-d3d2-4118-8f54-eb36c5f29983" volumeName="kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089146    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f923698-d3d2-4118-8f54-eb36c5f29983" volumeName="kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089165    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f923698-d3d2-4118-8f54-eb36c5f29983" volumeName="kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089186    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f923698-d3d2-4118-8f54-eb36c5f29983" volumeName="kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089205    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="358ed64f-afa7-41b0-9e8f-d981bebf2750" volumeName="kubernetes.io/projected/358ed64f-afa7-41b0-9e8f-d981bebf2750-kube-api-access-5wzzd" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089225    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8522b9-5150-4e60-95bb-f9b04a71add9" volumeName="kubernetes.io/projected/4a8522b9-5150-4e60-95bb-f9b04a71add9-kube-api-access-czdkb" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089242    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab9d5e-f98a-4b5a-b72b-175e9182f488" volumeName="kubernetes.io/projected/70ab9d5e-f98a-4b5a-b72b-175e9182f488-kube-api-access-g9wvp" seLinuxMountContext=""
Nov 03 11:50:51.089264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089260    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="358ed64f-afa7-41b0-9e8f-d981bebf2750" volumeName="kubernetes.io/secret/358ed64f-afa7-41b0-9e8f-d981bebf2750-cert" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089277    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77d106d7-59a7-459c-b012-aaa087cb50d9" volumeName="kubernetes.io/secret/77d106d7-59a7-459c-b012-aaa087cb50d9-agent-certs" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089307    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef83543a-8e80-486b-adeb-4d5aad83d4b0" volumeName="kubernetes.io/projected/ef83543a-8e80-486b-adeb-4d5aad83d4b0-kube-api-access-z4gzp" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089326    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874b36e5-1db3-498f-a51c-e23ccfb5947b" volumeName="kubernetes.io/secret/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089341    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e19744a-acf7-4e34-957f-c01efa20f0b2" volumeName="kubernetes.io/projected/9e19744a-acf7-4e34-957f-c01efa20f0b2-kube-api-access-wkfk8" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089359    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a208df1b-dba2-4b61-bd72-d7a4dffcada0" volumeName="kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089375    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a208df1b-dba2-4b61-bd72-d7a4dffcada0" volumeName="kubernetes.io/projected/a208df1b-dba2-4b61-bd72-d7a4dffcada0-kube-api-access-5mn4m" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089389    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8" volumeName="kubernetes.io/configmap/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-telemetry-config" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089409    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d378c41e-4f5f-4f08-b051-23694fee59d4" volumeName="kubernetes.io/projected/d378c41e-4f5f-4f08-b051-23694fee59d4-kube-api-access-nnq6q" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089428    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5daea77e-b365-4fc5-a5f3-bd9ad7c615e6" volumeName="kubernetes.io/configmap/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-iptables-alerter-script" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089444    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab9d5e-f98a-4b5a-b72b-175e9182f488" volumeName="kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-trusted-ca" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089512    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab9d5e-f98a-4b5a-b72b-175e9182f488" volumeName="kubernetes.io/secret/70ab9d5e-f98a-4b5a-b72b-175e9182f488-serving-cert" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089534    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874b36e5-1db3-498f-a51c-e23ccfb5947b" volumeName="kubernetes.io/projected/874b36e5-1db3-498f-a51c-e23ccfb5947b-kube-api-access-ccdzg" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089553    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfd61052-ba77-4b91-bb1b-9d808791b96b" volumeName="kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-tmp" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089574    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3" volumeName="kubernetes.io/projected/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3-kube-api-access-z7hjg" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089593    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874b36e5-1db3-498f-a51c-e23ccfb5947b" volumeName="kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-env-overrides" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089612    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e156d6b-a0b5-460d-8946-7f7329753e54" volumeName="kubernetes.io/projected/8e156d6b-a0b5-460d-8946-7f7329753e54-kube-api-access-gmlvw" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089629    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab06cb5-f005-4c43-a247-5f38134286cb" volumeName="kubernetes.io/configmap/bab06cb5-f005-4c43-a247-5f38134286cb-config" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089646    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef83543a-8e80-486b-adeb-4d5aad83d4b0" volumeName="kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-snapshots" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089664    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188932f7-09f2-4a38-94ec-8db245a5c89a" volumeName="kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089680    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71b27f5e-a0e1-4ba7-a4af-8282a4639d41" volumeName="kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-default-certificate" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089710    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71b27f5e-a0e1-4ba7-a4af-8282a4639d41" volumeName="kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-stats-auth" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089724    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfd61052-ba77-4b91-bb1b-9d808791b96b" volumeName="kubernetes.io/projected/bfd61052-ba77-4b91-bb1b-9d808791b96b-kube-api-access-smlbr" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089740    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8" volumeName="kubernetes.io/projected/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-kube-api-access-lq7m9" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089760    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f115611-61eb-45bb-9e13-34b7a2519dd5" volumeName="kubernetes.io/projected/4f115611-61eb-45bb-9e13-34b7a2519dd5-kube-api-access-bxxks" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089774    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5daea77e-b365-4fc5-a5f3-bd9ad7c615e6" volumeName="kubernetes.io/projected/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-kube-api-access-j9t6r" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089794    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7184fff6-b600-467a-bd3b-f11fd6f6558f" volumeName="kubernetes.io/projected/7184fff6-b600-467a-bd3b-f11fd6f6558f-kube-api-access-7tzvz" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089811    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c1607bc-d5e4-46b1-be5e-e27e7462a967" volumeName="kubernetes.io/secret/8c1607bc-d5e4-46b1-be5e-e27e7462a967-serving-cert" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089828    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be887ffe-4891-4231-93ee-13d03c4efd85" volumeName="kubernetes.io/configmap/be887ffe-4891-4231-93ee-13d03c4efd85-nginx-conf" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089843    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f923698-d3d2-4118-8f54-eb36c5f29983" volumeName="kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates" seLinuxMountContext=""
Nov 03 11:50:51.089869 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089860    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188932f7-09f2-4a38-94ec-8db245a5c89a" volumeName="kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted" seLinuxMountContext=""
Nov 03 11:50:51.090775 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089879    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab06cb5-f005-4c43-a247-5f38134286cb" volumeName="kubernetes.io/projected/bab06cb5-f005-4c43-a247-5f38134286cb-kube-api-access-qwz5v" seLinuxMountContext=""
Nov 03 11:50:51.090775 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089912    4909 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1" volumeName="kubernetes.io/projected/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-kube-api-access-zxpx9" seLinuxMountContext=""
Nov 03 11:50:51.090775 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089925    4909 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 11:50:51.090775 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.089944    4909 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 11:50:51.093186 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.093167    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/1.log"
Nov 03 11:50:51.097453 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.097416    4909 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 11:50:51.103205 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.102907    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/1.log"
Nov 03 11:50:51.115384 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.115362    4909 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 11:50:51.115384 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.115385    4909 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 11:50:51.115509 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.115399    4909 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 11:50:51.115509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.115430    4909 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 11:50:51.128881 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.128854    4909 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="1a74c97683e5e5ca1aa9360db74296aa65de9f97127622f07482902310e3d9b6" exitCode=0
Nov 03 11:50:51.128951 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.128881    4909 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="736cd7fd5bd1bba313b7996f2fd3372a94446e7b49d3e644ccdb8aa23b390417" exitCode=0
Nov 03 11:50:51.128951 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.128894    4909 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="4c7a9dc7cd00fbb38aca0e093df6c44a41a19b790216889dbd2e06f37b8545fb" exitCode=0
Nov 03 11:50:51.128951 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.128902    4909 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="9fcfb324eacf82d2cb783f32e95f4b67ab76d21b00a513c241c833626302cd4e" exitCode=0
Nov 03 11:50:51.128951 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.128911    4909 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="af15450a49a195894df480790c87b85a88c0e8b8f705ff654a3adaa9ed5fbb83" exitCode=0
Nov 03 11:50:51.128951 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.128919    4909 generic.go:356] "Generic (PLEG): container finished" podID="a208df1b-dba2-4b61-bd72-d7a4dffcada0" containerID="95d471dc74808ef65b660e60af940711015f9064bae89e67cd10e49e01fc1828" exitCode=0
Nov 03 11:50:51.133176 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.132788    4909 generic.go:356] "Generic (PLEG): container finished" podID="b495a64a0efaf6f05f21f42ec740163f" containerID="af2a3b42bc6fbf04a935dcba5ed69d97ae2f69eea46c2aa86584475750122cd9" exitCode=0
Nov 03 11:50:51.153852 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.153836    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/1.log"
Nov 03 11:50:51.153921 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.153859    4909 generic.go:356] "Generic (PLEG): container finished" podID="70ab9d5e-f98a-4b5a-b72b-175e9182f488" containerID="16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454" exitCode=255
Nov 03 11:50:51.172422 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.172405    4909 manager.go:324] Recovery completed
Nov 03 11:50:51.185229 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.185172    4909 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.us-west-2.compute.internal\" not found"
Nov 03 11:50:51.191316 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.191300    4909 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:51.196393 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.196374    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:51.196475 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.196409    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:51.196475 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.196424    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:51.198343 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198327    4909 cpu_manager.go:226] "Starting" policy="none"
Nov 03 11:50:51.198343 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198341    4909 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 11:50:51.198477 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198342    4909 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:51.198477 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198359    4909 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 11:50:51.198552 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198501    4909 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 11:50:51.198552 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198515    4909 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 11:50:51.198552 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198546    4909 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 11:50:51.198552 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198552    4909 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 11:50:51.198658 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198557    4909 policy_none.go:50] "Start"
Nov 03 11:50:51.198658 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198566    4909 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 11:50:51.198658 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.198576    4909 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 11:50:51.199156 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.199143    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:51.199199 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.199170    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:51.199199 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.199183    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:51.202020 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.202006    4909 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 11:50:51.202065 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.202021    4909 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 11:50:51.202065 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.202027    4909 policy_none.go:44] "Start"
Nov 03 11:50:51.206294 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.206268    4909 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 11:50:51.209132 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209121    4909 manager.go:341] "Starting Device Plugin manager"
Nov 03 11:50:51.209195 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.209145    4909 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 11:50:51.209195 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209154    4909 server.go:87] "Starting device plugin registration server"
Nov 03 11:50:51.209304 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209295    4909 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 11:50:51.209347 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209304    4909 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 11:50:51.209415 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209397    4909 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 11:50:51.209534 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209500    4909 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 11:50:51.209622 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209545    4909 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 11:50:51.209622 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209552    4909 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 11:50:51.210118 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.209603    4909 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T11:50:51.209529784Z","UUID":"e5b2c4fa-815a-442b-931f-0c9985ea1581","Handler":null,"Name":"","Endpoint":""}
Nov 03 11:50:51.210911 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.210893    4909 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 11:50:51.211039 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.211020    4909 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-136.us-west-2.compute.internal\" not found"
Nov 03 11:50:51.211307 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.211288    4909 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 11:50:51.211397 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.211315    4909 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 11:50:51.212382 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.212360    4909 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:51.212527 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.212507    4909 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:51.215530 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.215500    4909 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"]
Nov 03 11:50:51.215630 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.215601    4909 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:51.216358 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.216342    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:51.216429 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.216373    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:51.216429 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.216387    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:51.217276 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.217258    4909 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:51.217385 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.217366    4909 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:51.219326 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219308    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:51.219326 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219317    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:51.219452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219338    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:51.219452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219349    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:51.219452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219357    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:51.219452 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219364    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:51.219604 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.219520    4909 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.us-west-2.compute.internal\" not found" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.219808 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219792    4909 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:51.219877 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219824    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"b495a64a0efaf6f05f21f42ec740163f","Type":"ContainerStarted","Data":"d791d084baefd4e028ae33c54dc6e0c8573fb3ef8ed4798c095fa083598bf749"}
Nov 03 11:50:51.219928 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219899    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"b495a64a0efaf6f05f21f42ec740163f","Type":"ContainerDied","Data":"af2a3b42bc6fbf04a935dcba5ed69d97ae2f69eea46c2aa86584475750122cd9"}
Nov 03 11:50:51.219984 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.219944    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"b495a64a0efaf6f05f21f42ec740163f","Type":"ContainerStarted","Data":"b69d207f2ab269a926237b66e0d6c6d620f87436968868dd74649a18890fb0b0"}
Nov 03 11:50:51.220054 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.220040    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"f2905ce8bf73baad4434061659facba8","Type":"ContainerStarted","Data":"3a22d380c8920eaf8c80fc2ac371118649209bb665d3855a43f72ee0db7e6305"}
Nov 03 11:50:51.220102 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.220065    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal" event={"ID":"f2905ce8bf73baad4434061659facba8","Type":"ContainerStarted","Data":"3225e186e6af2ef90eb963872b3993335886c0477b427b6ddafb93319ae3c596"}
Nov 03 11:50:51.220438 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.220420    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:51.220502 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.220451    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:51.220502 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.220464    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:51.220585 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.220570    4909 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.us-west-2.compute.internal\" not found" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.254478 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.254455    4909 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:51.284720 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.284678    4909 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.299035 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.299014    4909 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 11:50:51.304675 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.304660    4909 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:51.309920 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.309908    4909 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:51.310657 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.310641    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:51.310741 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.310672    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:51.310741 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.310687    4909 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:51.310856 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.310750    4909 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.344632 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.344607    4909 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:51.344733 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.344653    4909 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.344733 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.344668    4909 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.399967 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.399945    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.400043 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.399969    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.400043 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.399986    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2905ce8bf73baad4434061659facba8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"f2905ce8bf73baad4434061659facba8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.400154 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.400140    4909 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:51.400194 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:51.400173    4909 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.410748 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.410732    4909 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.410797 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.410780    4909 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.500581 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.500560    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2905ce8bf73baad4434061659facba8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"f2905ce8bf73baad4434061659facba8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.500693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.500590    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.500693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.500609    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.500693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.500647    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.500693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.500665    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b495a64a0efaf6f05f21f42ec740163f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"b495a64a0efaf6f05f21f42ec740163f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:51.500693 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:51.500670    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2905ce8bf73baad4434061659facba8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" (UID: \"f2905ce8bf73baad4434061659facba8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:52.059458 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.059431    4909 apiserver.go:52] "Watching apiserver"
Nov 03 11:50:52.164836 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.164810    4909 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:52.164972 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.164904    4909 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:52.201253 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.201217    4909 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 11:50:52.201934 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.201908    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xv5pl","openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh","kube-system/konnectivity-agent-df7t5","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p","openshift-insights/insights-runtime-extractor-bq6v7","openshift-ovn-kubernetes/ovnkube-node-5qjf7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg","openshift-image-registry/image-registry-b9dcd8545-gs4r8","openshift-multus/multus-additional-cni-plugins-j7bdh","openshift-cluster-node-tuning-operator/tuned-742vk","openshift-dns/dns-default-w7l5k","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb","kube-system/global-pull-secret-syncer-lsvz7","openshift-console-operator/console-operator-7cd56b8dbc-rmmzr","openshift-network-operator/iptables-alerter-8pdls","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m","openshift-ingress-canary/ingress-canary-7j2qr","openshift-multus/multus-dqh56","kube-system/kas-connection-checker-685895c7c9-n7zd8","openshift-image-registry/node-ca-dcljv","openshift-ingress/router-default-b5bc54f87-gkx8h","openshift-insights/insights-operator-7c4986d449-8cwrc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t","openshift-network-diagnostics/network-check-target-gkt2k","openshift-multus/network-metrics-daemon-cs8vd","kube-system/kas-connection-checker-685895c7c9-nct5z","openshift-dns/node-resolver-7bdv2","kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l","openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph","openshift-image-registry/image-registry-6796696b5b-mlp66","openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"]
Nov 03 11:50:52.203364 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.203320    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.206928 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.206901    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.207022 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.206939    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.208066 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.208042    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.208943 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.208927    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:52.213294 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.213278    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:52.215165 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.215148    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.215779 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.215762    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.217685 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.217665    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.220965 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.220950    4909 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:52.221022 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.220999    4909 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:52.221449 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.221438    4909 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:52.221493 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.221464    4909 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.us-west-2.compute.internal"
Nov 03 11:50:52.252540 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252519    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 11:50:52.252622 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252576    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.252622 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252594    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.252622 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252596    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 11:50:52.252791 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252594    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.252791 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252653    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.252791 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252716    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 11:50:52.252791 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252782    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 11:50:52.252968 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252860    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 11:50:52.252968 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252888    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.252968 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252912    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 11:50:52.252968 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252944    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.253142 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.252991    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.253389 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.253371    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.253808 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.253795    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 11:50:52.255554 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.255540    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 11:50:52.255899 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.255884    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 11:50:52.256052 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256038    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 11:50:52.256114 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256102    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.256412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256399    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 11:50:52.256875 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256856    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256862    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256888    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256905    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256914    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256923    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256934    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256952    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256962    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.257007 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.256962    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.257379 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.257013    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.257379 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.257277    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.287621 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.287601    4909 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 11:50:52.299006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.298949    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 11:50:52.302073 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302045    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 11:50:52.302171 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302148    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.302256 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302234    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 11:50:52.302343 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302320    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-65pss\""
Nov 03 11:50:52.302404 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302342    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 11:50:52.302404 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302386    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 11:50:52.302503 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302403    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 11:50:52.302503 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302422    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2ks4w\""
Nov 03 11:50:52.302503 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302322    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 11:50:52.302503 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302492    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.302751 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302527    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jp68j\""
Nov 03 11:50:52.302751 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302572    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 11:50:52.302751 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302625    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-swh4m\""
Nov 03 11:50:52.303070 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302758    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.303070 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302766    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.303070 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302894    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.303070 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302903    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xf5fg\""
Nov 03 11:50:52.303070 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302987    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 11:50:52.303070 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.302997    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 11:50:52.303070 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303036    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.303500 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303171    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 11:50:52.303500 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303250    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxdwl\""
Nov 03 11:50:52.303500 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303322    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.303500 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303380    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 11:50:52.303500 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303467    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 11:50:52.303777 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303541    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.303777 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303589    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-whvfc\""
Nov 03 11:50:52.303777 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303604    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 11:50:52.303777 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303639    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.303777 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303679    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303783    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303836    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303845    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303894    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303908    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303880    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303917    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303971    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.304044 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.303987    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.304404 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.304170    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.304925 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.304905    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.305033 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.304936    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlbr\" (UniqueName: \"kubernetes.io/projected/bfd61052-ba77-4b91-bb1b-9d808791b96b-kube-api-access-smlbr\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.305033 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.304945    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.305033 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.304956    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gzp\" (UniqueName: \"kubernetes.io/projected/ef83543a-8e80-486b-adeb-4d5aad83d4b0-kube-api-access-z4gzp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.305033 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.304974    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 11:50:52.305033 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305002    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 11:50:52.305033 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305013    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bab06cb5-f005-4c43-a247-5f38134286cb-serving-cert\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305064    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305075    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305081    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305154    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305188    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305222    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305227    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tvsxn\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305231    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305243    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bab06cb5-f005-4c43-a247-5f38134286cb-serving-cert\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305262    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305270    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g5g7l\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305302    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305156    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305313    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vn7bf\""
Nov 03 11:50:52.305330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305321    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6gwtx\""
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305346    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-stats-auth\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305367    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305388    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-script-lib\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305471    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-system-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305524    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305544    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305578    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef83543a-8e80-486b-adeb-4d5aad83d4b0-serving-cert\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305603    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jp4k\""
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305606    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz5v\" (UniqueName: \"kubernetes.io/projected/bab06cb5-f005-4c43-a247-5f38134286cb-kube-api-access-qwz5v\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305636    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/77d106d7-59a7-459c-b012-aaa087cb50d9-agent-certs\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305549    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fw7f8\""
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305660    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-tmp-dir\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305688    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-modprobe-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305716    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzgk2\""
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305730    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-var-lib-kubelet\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305650    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-stats-auth\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305755    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305783    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-cni-binary-copy\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305801    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-script-lib\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305806    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-kubelet\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305836    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef83543a-8e80-486b-adeb-4d5aad83d4b0-serving-cert\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305842    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-tmp-dir\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305887    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8522b9-5150-4e60-95bb-f9b04a71add9-tmp-dir\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305935    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d378c41e-4f5f-4f08-b051-23694fee59d4-host\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:52.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.305964    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-cnibin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306007    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306040    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8522b9-5150-4e60-95bb-f9b04a71add9-tmp-dir\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306045    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-log-socket\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306093    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-systemd\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306111    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/098926dd-2788-41f4-a964-5f7448c8c41f-original-pull-secret\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306134    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-kubelet-config\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306141    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm8xm\""
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306172    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxks\" (UniqueName: \"kubernetes.io/projected/4f115611-61eb-45bb-9e13-34b7a2519dd5-kube-api-access-bxxks\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306140    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-cni-binary-copy\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306235    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-config\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306316    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306351    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-daemon-config\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306371    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-netns\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306387    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eccf3810-774d-4fe5-9516-6c08440c1505-crio-socket\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306409    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306424    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306440    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-config\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306441    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxx4d\" (UniqueName: \"kubernetes.io/projected/eccf3810-774d-4fe5-9516-6c08440c1505-kube-api-access-jxx4d\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306492    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-multus-certs\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306528    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306531    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-daemon-config\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306530    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nf6r\" (UniqueName: \"kubernetes.io/projected/8c1607bc-d5e4-46b1-be5e-e27e7462a967-kube-api-access-7nf6r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306670    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8522b9-5150-4e60-95bb-f9b04a71add9-config-volume\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306717    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hsb\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306764    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxwp\" (UniqueName: \"kubernetes.io/projected/b9319607-a6c8-4003-89bd-14da12ba17a8-kube-api-access-hzxwp\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306788    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdkb\" (UniqueName: \"kubernetes.io/projected/4a8522b9-5150-4e60-95bb-f9b04a71add9-kube-api-access-czdkb\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306813    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306840    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-kubernetes\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306860    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-hosts-file\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306886    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkfhq\" (UniqueName: \"kubernetes.io/projected/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-kube-api-access-gkfhq\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.306871 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306888    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8522b9-5150-4e60-95bb-f9b04a71add9-config-volume\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306937    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-dbus\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306959    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-netd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.306990    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-trusted-ca\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307033    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzzd\" (UniqueName: \"kubernetes.io/projected/358ed64f-afa7-41b0-9e8f-d981bebf2750-kube-api-access-5wzzd\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307059    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307082    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-tmp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307106    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-tmp\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307124    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307160    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307186    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307207    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-tmp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307212    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-tmp\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307212    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-k8s-cni-cncf-io\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307259    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-multus\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307290    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307310    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70ab9d5e-f98a-4b5a-b72b-175e9182f488-trusted-ca\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307377    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307391    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-systemd-units\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307411    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-systemd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307425    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307447    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mn4m\" (UniqueName: \"kubernetes.io/projected/a208df1b-dba2-4b61-bd72-d7a4dffcada0-kube-api-access-5mn4m\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307486    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d378c41e-4f5f-4f08-b051-23694fee59d4-serviceca\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307542    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307605    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkkf\" (UniqueName: \"kubernetes.io/projected/7b7eef80-4dd6-4087-b74c-66adfb323841-kube-api-access-xqkkf\") pod \"kas-connection-checker-685895c7c9-n7zd8\" (UID: \"7b7eef80-4dd6-4087-b74c-66adfb323841\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zd8"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307711    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncm5f\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307732    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d378c41e-4f5f-4f08-b051-23694fee59d4-serviceca\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307834    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-conf\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307876    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-etc-kubernetes\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307927    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-slash\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.307955    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-env-overrides\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308019    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/77d106d7-59a7-459c-b012-aaa087cb50d9-konnectivity-ca\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308076    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-env-overrides\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308095    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308130    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-node-log\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308156    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-config\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.308164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308180    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/77d106d7-59a7-459c-b012-aaa087cb50d9-konnectivity-ca\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308192    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxpx9\" (UniqueName: \"kubernetes.io/projected/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-kube-api-access-zxpx9\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308274    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308320    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqw5\" (UniqueName: \"kubernetes.io/projected/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-kube-api-access-xkqw5\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308356    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovnkube-config\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308368    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308392    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovn-node-metrics-cert\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308421    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308440    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cnibin\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308462    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308491    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-host-slash\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308506    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308518    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b36e5-1db3-498f-a51c-e23ccfb5947b-ovn-node-metrics-cert\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308590    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-etc-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308615    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-registration-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308644    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308675    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-sys\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308691    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-host\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308757    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308778    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308799    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308817    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a208df1b-dba2-4b61-bd72-d7a4dffcada0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308823    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308888    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.308987    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309033    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wvp\" (UniqueName: \"kubernetes.io/projected/70ab9d5e-f98a-4b5a-b72b-175e9182f488-kube-api-access-g9wvp\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309080    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309132    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309156    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bab06cb5-f005-4c43-a247-5f38134286cb-config\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309264    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmlvw\" (UniqueName: \"kubernetes.io/projected/8e156d6b-a0b5-460d-8946-7f7329753e54-kube-api-access-gmlvw\") pod \"kas-connection-checker-685895c7c9-xv5pl\" (UID: \"8e156d6b-a0b5-460d-8946-7f7329753e54\") " pod="kube-system/kas-connection-checker-685895c7c9-xv5pl"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309308    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309409    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bab06cb5-f005-4c43-a247-5f38134286cb-config\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309418    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-bin\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309443    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309463    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c1607bc-d5e4-46b1-be5e-e27e7462a967-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309497    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309529    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309543    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309574    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c1607bc-d5e4-46b1-be5e-e27e7462a967-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309576    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.309689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309603    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309737    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-var-lib-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309753    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309757    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-iptables-alerter-script\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309786    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-snapshots\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309818    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309876    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309888    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-socket-dir-parent\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309914    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef83543a-8e80-486b-adeb-4d5aad83d4b0-snapshots\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309922    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-conf-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309965    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-ovn\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309976    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-iptables-alerter-script\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309992    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.309994    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310011    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310035    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-netns\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310064    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-device-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310105    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310132    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-system-cni-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310151    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310175    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-hostroot\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310197    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eccf3810-774d-4fe5-9516-6c08440c1505-data-volume\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310227    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysconfig\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310257    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eccf3810-774d-4fe5-9516-6c08440c1505-data-volume\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310298    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310330    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktl5\" (UniqueName: \"kubernetes.io/projected/9ceda7ca-9770-42ec-973a-0435ff74dd29-kube-api-access-lktl5\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310361    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310377    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7m9\" (UniqueName: \"kubernetes.io/projected/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-kube-api-access-lq7m9\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310402    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310500    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-lib-modules\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310505    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310547    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-bin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310579    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70ab9d5e-f98a-4b5a-b72b-175e9182f488-serving-cert\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310615    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hjg\" (UniqueName: \"kubernetes.io/projected/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3-kube-api-access-z7hjg\") pod \"migrator-66df5d496b-g4hph\" (UID: \"85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310637    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-default-certificate\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310653    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358ed64f-afa7-41b0-9e8f-d981bebf2750-cert\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310667    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be887ffe-4891-4231-93ee-13d03c4efd85-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310685    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70ab9d5e-f98a-4b5a-b72b-175e9182f488-serving-cert\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310691    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-tuned\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310725    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq6q\" (UniqueName: \"kubernetes.io/projected/d378c41e-4f5f-4f08-b051-23694fee59d4-kube-api-access-nnq6q\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310817    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxzb\" (UniqueName: \"kubernetes.io/projected/4e9fac69-3de2-419e-931f-f1cb74a8aebb-kube-api-access-4lxzb\") pod \"volume-data-source-validator-58c9f5f46d-ldg6p\" (UID: \"4e9fac69-3de2-419e-931f-f1cb74a8aebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310828    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-tuned\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310847    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-socket-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310870    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-default-certificate\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310878    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdzg\" (UniqueName: \"kubernetes.io/projected/874b36e5-1db3-498f-a51c-e23ccfb5947b-kube-api-access-ccdzg\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310910    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c1607bc-d5e4-46b1-be5e-e27e7462a967-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310964    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be887ffe-4891-4231-93ee-13d03c4efd85-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310963    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.310964    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358ed64f-afa7-41b0-9e8f-d981bebf2750-cert\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311000    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-sys-fs\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.311006 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311026    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-run\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311060    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9t6r\" (UniqueName: \"kubernetes.io/projected/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-kube-api-access-j9t6r\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311090    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311117    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-service-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311146    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c1607bc-d5e4-46b1-be5e-e27e7462a967-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311151    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311209    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-kubelet\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311255    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef83543a-8e80-486b-adeb-4d5aad83d4b0-service-ca-bundle\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311260    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzvz\" (UniqueName: \"kubernetes.io/projected/7184fff6-b600-467a-bd3b-f11fd6f6558f-kube-api-access-7tzvz\") pod \"network-check-source-6f78d59f99-fxq8l\" (UID: \"7184fff6-b600-467a-bd3b-f11fd6f6558f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"
Nov 03 11:50:52.312412 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.311295    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfk8\" (UniqueName: \"kubernetes.io/projected/9e19744a-acf7-4e34-957f-c01efa20f0b2-kube-api-access-wkfk8\") pod \"kas-connection-checker-685895c7c9-nct5z\" (UID: \"9e19744a-acf7-4e34-957f-c01efa20f0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-nct5z"
Nov 03 11:50:52.316489 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.316474    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/77d106d7-59a7-459c-b012-aaa087cb50d9-agent-certs\") pod \"konnectivity-agent-df7t5\" (UID: \"77d106d7-59a7-459c-b012-aaa087cb50d9\") " pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:50:52.320305 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.320282    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 11:50:52.320381 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.320355    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7npm\""
Nov 03 11:50:52.320381 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.320359    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xf9nj\""
Nov 03 11:50:52.320454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.320405    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nxjhp\""
Nov 03 11:50:52.364142 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.364122    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdkb\" (UniqueName: \"kubernetes.io/projected/4a8522b9-5150-4e60-95bb-f9b04a71add9-kube-api-access-czdkb\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.364532 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.364516    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nf6r\" (UniqueName: \"kubernetes.io/projected/8c1607bc-d5e4-46b1-be5e-e27e7462a967-kube-api-access-7nf6r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m\" (UID: \"8c1607bc-d5e4-46b1-be5e-e27e7462a967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m"
Nov 03 11:50:52.365036 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.365021    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlbr\" (UniqueName: \"kubernetes.io/projected/bfd61052-ba77-4b91-bb1b-9d808791b96b-kube-api-access-smlbr\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.365223 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.365209    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxx4d\" (UniqueName: \"kubernetes.io/projected/eccf3810-774d-4fe5-9516-6c08440c1505-kube-api-access-jxx4d\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.365563 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.365548    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxpx9\" (UniqueName: \"kubernetes.io/projected/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-kube-api-access-zxpx9\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:52.365593 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.365574    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfk8\" (UniqueName: \"kubernetes.io/projected/9e19744a-acf7-4e34-957f-c01efa20f0b2-kube-api-access-wkfk8\") pod \"kas-connection-checker-685895c7c9-nct5z\" (UID: \"9e19744a-acf7-4e34-957f-c01efa20f0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-nct5z"
Nov 03 11:50:52.366147 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.366127    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdzg\" (UniqueName: \"kubernetes.io/projected/874b36e5-1db3-498f-a51c-e23ccfb5947b-kube-api-access-ccdzg\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.366244 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.366227    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9t6r\" (UniqueName: \"kubernetes.io/projected/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-kube-api-access-j9t6r\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:52.366737 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.366722    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hsb\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.371644 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.371611    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqw5\" (UniqueName: \"kubernetes.io/projected/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-kube-api-access-xkqw5\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:52.371760 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.371652    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxwp\" (UniqueName: \"kubernetes.io/projected/b9319607-a6c8-4003-89bd-14da12ba17a8-kube-api-access-hzxwp\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.371760 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.371718    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.371760 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.371734    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mn4m\" (UniqueName: \"kubernetes.io/projected/a208df1b-dba2-4b61-bd72-d7a4dffcada0-kube-api-access-5mn4m\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.371760 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.371720    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzzd\" (UniqueName: \"kubernetes.io/projected/358ed64f-afa7-41b0-9e8f-d981bebf2750-kube-api-access-5wzzd\") pod \"ingress-canary-7j2qr\" (UID: \"358ed64f-afa7-41b0-9e8f-d981bebf2750\") " pod="openshift-ingress-canary/ingress-canary-7j2qr"
Nov 03 11:50:52.372091 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.371873    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkfhq\" (UniqueName: \"kubernetes.io/projected/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-kube-api-access-gkfhq\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.372091 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.371929    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzvz\" (UniqueName: \"kubernetes.io/projected/7184fff6-b600-467a-bd3b-f11fd6f6558f-kube-api-access-7tzvz\") pod \"network-check-source-6f78d59f99-fxq8l\" (UID: \"7184fff6-b600-467a-bd3b-f11fd6f6558f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fxq8l"
Nov 03 11:50:52.372173 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372152    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkkf\" (UniqueName: \"kubernetes.io/projected/7b7eef80-4dd6-4087-b74c-66adfb323841-kube-api-access-xqkkf\") pod \"kas-connection-checker-685895c7c9-n7zd8\" (UID: \"7b7eef80-4dd6-4087-b74c-66adfb323841\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zd8"
Nov 03 11:50:52.372396 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372376    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxks\" (UniqueName: \"kubernetes.io/projected/4f115611-61eb-45bb-9e13-34b7a2519dd5-kube-api-access-bxxks\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.372462 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372440    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktl5\" (UniqueName: \"kubernetes.io/projected/9ceda7ca-9770-42ec-973a-0435ff74dd29-kube-api-access-lktl5\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.372526 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372471    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncm5f\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.372650 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372631    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wvp\" (UniqueName: \"kubernetes.io/projected/70ab9d5e-f98a-4b5a-b72b-175e9182f488-kube-api-access-g9wvp\") pod \"console-operator-7cd56b8dbc-rmmzr\" (UID: \"70ab9d5e-f98a-4b5a-b72b-175e9182f488\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:52.372683 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372660    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq6q\" (UniqueName: \"kubernetes.io/projected/d378c41e-4f5f-4f08-b051-23694fee59d4-kube-api-access-nnq6q\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:52.372786 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372660    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gzp\" (UniqueName: \"kubernetes.io/projected/ef83543a-8e80-486b-adeb-4d5aad83d4b0-kube-api-access-z4gzp\") pod \"insights-operator-7c4986d449-8cwrc\" (UID: \"ef83543a-8e80-486b-adeb-4d5aad83d4b0\") " pod="openshift-insights/insights-operator-7c4986d449-8cwrc"
Nov 03 11:50:52.372847 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372829    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.373018 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.372999    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz5v\" (UniqueName: \"kubernetes.io/projected/bab06cb5-f005-4c43-a247-5f38134286cb-kube-api-access-qwz5v\") pod \"service-ca-operator-544d749c8c-5t86t\" (UID: \"bab06cb5-f005-4c43-a247-5f38134286cb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t"
Nov 03 11:50:52.373155 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.373137    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmlvw\" (UniqueName: \"kubernetes.io/projected/8e156d6b-a0b5-460d-8946-7f7329753e54-kube-api-access-gmlvw\") pod \"kas-connection-checker-685895c7c9-xv5pl\" (UID: \"8e156d6b-a0b5-460d-8946-7f7329753e54\") " pod="kube-system/kas-connection-checker-685895c7c9-xv5pl"
Nov 03 11:50:52.373194 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.373183    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7m9\" (UniqueName: \"kubernetes.io/projected/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-kube-api-access-lq7m9\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.373231 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.373191    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hjg\" (UniqueName: \"kubernetes.io/projected/85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3-kube-api-access-z7hjg\") pod \"migrator-66df5d496b-g4hph\" (UID: \"85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph"
Nov 03 11:50:52.373337 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.373322    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxzb\" (UniqueName: \"kubernetes.io/projected/4e9fac69-3de2-419e-931f-f1cb74a8aebb-kube-api-access-4lxzb\") pod \"volume-data-source-validator-58c9f5f46d-ldg6p\" (UID: \"4e9fac69-3de2-419e-931f-f1cb74a8aebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ldg6p"
Nov 03 11:50:52.373456 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.373441    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pcf\" (UniqueName: \"kubernetes.io/projected/ad5bba67-4cce-447f-aa79-d6061b83fb63-kube-api-access-g9pcf\") pod \"network-check-target-gkt2k\" (UID: \"ad5bba67-4cce-447f-aa79-d6061b83fb63\") " pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:52.412004 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.411985    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-kubernetes\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412092 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412010    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-hosts-file\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:52.412092 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412027    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-dbus\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.412092 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412041    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-netd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412092 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412060    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-k8s-cni-cncf-io\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412091    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-multus\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412107    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-k8s-cni-cncf-io\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412110    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2-hosts-file\") pod \"node-resolver-7bdv2\" (UID: \"cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2\") " pod="openshift-dns/node-resolver-7bdv2"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412119    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-systemd-units\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412122    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-kubernetes\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412142    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-systemd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412147    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-netd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412167    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-dbus\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412182    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-multus\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412216    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-systemd-units\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412235    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.412249 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412252    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-conf\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412272    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-etc-kubernetes\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412295    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-slash\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412310    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412320    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-node-log\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412348    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412349    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-systemd\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412381    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-node-log\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412391    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412373    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412370    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-conf\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412398    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-etc-kubernetes\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412416    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-slash\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412436    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cnibin\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412450    4909 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412480    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-host-slash\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412485    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-cnibin\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412514    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.912495945 +0000 UTC m=+1.924962606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : secret "metrics-daemon-secret" not found
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412527    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5daea77e-b365-4fc5-a5f3-bd9ad7c615e6-host-slash\") pod \"iptables-alerter-8pdls\" (UID: \"5daea77e-b365-4fc5-a5f3-bd9ad7c615e6\") " pod="openshift-network-operator/iptables-alerter-8pdls"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412532    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412559    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-etc-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412602    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-registration-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412614    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-etc-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412617    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-sys\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412642    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-host\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412650    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-sys\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412657    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412671    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.912653885 +0000 UTC m=+1.925120543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412718    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-host\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412736    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.412752 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412761    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-bin\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412722    4909 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412786    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412811    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-cni-bin\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412718    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-registration-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412816    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.912806776 +0000 UTC m=+1.925273417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412855    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412857    4909 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412871    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412903    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.912893454 +0000 UTC m=+1.925360101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412809    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412929    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412946    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-var-lib-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412956    4909 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412965    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-socket-dir-parent\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.412980    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.912972492 +0000 UTC m=+1.925439133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412997    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-socket-dir-parent\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.412991    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-conf-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413022    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-var-lib-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413038    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-ovn\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413054    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413056    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-multus-conf-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413087    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-ovn\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413098    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-netns\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413125    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413136    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413139    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-device-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413154    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-system-cni-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413168    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413179    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.913166904 +0000 UTC m=+1.925633560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413195    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-run-netns\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413205    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-hostroot\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413208    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-system-cni-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413206    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-device-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413231    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysconfig\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413255    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413242    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-hostroot\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413275    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413306    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-lib-modules\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413321    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysctl-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413327    4909 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413345    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413350    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-sysconfig\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413363    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413348    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-bin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413379    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-cni-bin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413401    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.913387681 +0000 UTC m=+1.925854336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413415    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-lib-modules\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413434    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.913424918 +0000 UTC m=+1.925891560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413461    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-socket-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413480    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413495    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-sys-fs\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413515    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-run\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413540    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413564    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-kubelet\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413590    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413614    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413635    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-system-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413659    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-modprobe-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413680    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-var-lib-kubelet\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413724    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-kubelet\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413745    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d378c41e-4f5f-4f08-b051-23694fee59d4-host\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413761    4909 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413784    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-run-openvswitch\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.413812    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.913795287 +0000 UTC m=+1.926261946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413827    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-cnibin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.413812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413851    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-var-lib-kubelet\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413861    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d378c41e-4f5f-4f08-b051-23694fee59d4-host\") pod \"node-ca-dcljv\" (UID: \"d378c41e-4f5f-4f08-b051-23694fee59d4\") " pod="openshift-image-registry/node-ca-dcljv"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413765    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-cnibin\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413855    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-sys-fs\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413805    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-system-cni-dir\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413887    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-modprobe-d\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413893    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-log-socket\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413898    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413905    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-var-lib-kubelet\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413913    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9319607-a6c8-4003-89bd-14da12ba17a8-socket-dir\") pod \"aws-ebs-csi-driver-node-mrshg\" (UID: \"b9319607-a6c8-4003-89bd-14da12ba17a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrshg"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413918    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-systemd\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413853    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-run\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413940    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-host-kubelet\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413945    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/874b36e5-1db3-498f-a51c-e23ccfb5947b-log-socket\") pod \"ovnkube-node-5qjf7\" (UID: \"874b36e5-1db3-498f-a51c-e23ccfb5947b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413949    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a208df1b-dba2-4b61-bd72-d7a4dffcada0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7bdh\" (UID: \"a208df1b-dba2-4b61-bd72-d7a4dffcada0\") " pod="openshift-multus/multus-additional-cni-plugins-j7bdh"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413945    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-kubelet-config\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413981    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd61052-ba77-4b91-bb1b-9d808791b96b-etc-systemd\") pod \"tuned-742vk\" (UID: \"bfd61052-ba77-4b91-bb1b-9d808791b96b\") " pod="openshift-cluster-node-tuning-operator/tuned-742vk"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413982    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/098926dd-2788-41f4-a964-5f7448c8c41f-kubelet-config\") pod \"global-pull-secret-syncer-lsvz7\" (UID: \"098926dd-2788-41f4-a964-5f7448c8c41f\") " pod="kube-system/global-pull-secret-syncer-lsvz7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.413990    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-netns\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414018    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-netns\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414038    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eccf3810-774d-4fe5-9516-6c08440c1505-crio-socket\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414059    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414092    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eccf3810-774d-4fe5-9516-6c08440c1505-crio-socket\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414117    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414136    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-multus-certs\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414177    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ceda7ca-9770-42ec-973a-0435ff74dd29-host-run-multus-certs\") pod \"multus-dqh56\" (UID: \"9ceda7ca-9770-42ec-973a-0435ff74dd29\") " pod="openshift-multus/multus-dqh56"
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.414216    4909 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.414260    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.914250621 +0000 UTC m=+1.926717263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:52.415626 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.414422    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eccf3810-774d-4fe5-9516-6c08440c1505-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.503633 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.503600    4909 scope.go:122] "RemoveContainer" containerID="16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454"
Nov 03 11:50:52.518472 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.518455    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:50:52.918957 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.918882    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:52.918957 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.918919    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.918957 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.918937    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.918962    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.918981    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.919006    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.919025    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919038    4909 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919062    4909 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919050    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919031197 +0000 UTC m=+2.931497858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919095    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919101    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.91909065 +0000 UTC m=+2.931557294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : secret "metrics-daemon-secret" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919062    4909 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919116    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919109013 +0000 UTC m=+2.931575654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919108    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919139    4909 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919144    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919203    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:52.919211 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919147    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919127529 +0000 UTC m=+2.931594186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.919233    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919242    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919225228 +0000 UTC m=+2.931691883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919270    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919259764 +0000 UTC m=+2.931726418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919287    4909 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919297    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919287124 +0000 UTC m=+2.931753768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919320    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919310862 +0000 UTC m=+2.931777510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.919380    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:52.919442    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919482    4909 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919514    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.91950413 +0000 UTC m=+2.931970774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919549    4909 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:52.919862 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:52.919594    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:53.919582991 +0000 UTC m=+2.932049647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:53.170942 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.170874    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 11:50:53.171466 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.171288    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/1.log"
Nov 03 11:50:53.171466 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.171325    4909 generic.go:356] "Generic (PLEG): container finished" podID="70ab9d5e-f98a-4b5a-b72b-175e9182f488" containerID="33d6c027dcd7b7108e62bb871f2a46add2683efcb5fafaf5f13b08febbecb0a0" exitCode=255
Nov 03 11:50:53.171466 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.171393    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" event={"ID":"70ab9d5e-f98a-4b5a-b72b-175e9182f488","Type":"ContainerDied","Data":"33d6c027dcd7b7108e62bb871f2a46add2683efcb5fafaf5f13b08febbecb0a0"}
Nov 03 11:50:53.171466 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.171449    4909 scope.go:122] "RemoveContainer" containerID="16b3ad42dbdd96ee4f08cbab61b83bbb86dc55f277f5f8d82a31a7f186189454"
Nov 03 11:50:53.171920 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.171904    4909 scope.go:122] "RemoveContainer" containerID="33d6c027dcd7b7108e62bb871f2a46add2683efcb5fafaf5f13b08febbecb0a0"
Nov 03 11:50:53.172069 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.172050    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rmmzr_openshift-console-operator(70ab9d5e-f98a-4b5a-b72b-175e9182f488)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" podUID="70ab9d5e-f98a-4b5a-b72b-175e9182f488"
Nov 03 11:50:53.928851 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.928820    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:53.928851 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.928859    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.928879    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.928917    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.928967    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.928993    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929010    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929036    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929053    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929058    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929044425 +0000 UTC m=+4.941511079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.928998    4909 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929096    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929080684 +0000 UTC m=+4.941547339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929110    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929104359 +0000 UTC m=+4.941571000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:53.929111 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.929049    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.929144    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929112    4909 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.929171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929131    4909 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929216    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929206109 +0000 UTC m=+4.941672763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : secret "metrics-daemon-secret" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929178    4909 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929232    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.92922554 +0000 UTC m=+4.941692181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929244    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929239097 +0000 UTC m=+4.941705739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929269    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929254221 +0000 UTC m=+4.941720877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929279    4909 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929302    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929293761 +0000 UTC m=+4.941760443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.929320    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:53.929344    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929420    4909 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929456    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929447031 +0000 UTC m=+4.941913677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929456    4909 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:53.929605 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:53.929493    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:55.929484599 +0000 UTC m=+4.941951283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:54.178124 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:54.178097    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 11:50:54.179678 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:54.179626    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g4hph" event={"ID":"85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3","Type":"ContainerStarted","Data":"52efaa9de376ffeed973e16118d699fee2c534da3b6e2b2563c1dc5dded2a1d5"}
Nov 03 11:50:55.179166 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.179134    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:50:55.179565 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.179478    4909 scope.go:122] "RemoveContainer" containerID="33d6c027dcd7b7108e62bb871f2a46add2683efcb5fafaf5f13b08febbecb0a0"
Nov 03 11:50:55.179656 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.179636    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rmmzr_openshift-console-operator(70ab9d5e-f98a-4b5a-b72b-175e9182f488)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" podUID="70ab9d5e-f98a-4b5a-b72b-175e9182f488"
Nov 03 11:50:55.539819 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.539788    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:50:55.950056 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950027    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:55.950238 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950063    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:55.950238 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950090    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:55.950238 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950121    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:55.950238 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950151    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:55.950238 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950170    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:55.950238 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950189    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:55.950238 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950187    4909 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950250    4909 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950275    4909 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950282    4909 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950210    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950190    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950330    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950340    4909 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950295    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950275325 +0000 UTC m=+8.962741994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950371    4909 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950385    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950372553 +0000 UTC m=+8.962839206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950407    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950397016 +0000 UTC m=+8.962863663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950425    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert podName:be887ffe-4891-4231-93ee-13d03c4efd85 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950416164 +0000 UTC m=+8.962882808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d2xt7" (UID: "be887ffe-4891-4231-93ee-13d03c4efd85") : secret "networking-console-plugin-cert" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950439    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950432144 +0000 UTC m=+8.962898793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : secret "metrics-daemon-secret" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950461    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950452843 +0000 UTC m=+8.962919490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : secret "router-metrics-certs-default" not found
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950489    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:55.950509 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950520    4909 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 11:50:55.951024 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:55.950524    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:55.951024 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950547    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls podName:4f115611-61eb-45bb-9e13-34b7a2519dd5 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950536414 +0000 UTC m=+8.963003080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j2dmb" (UID: "4f115611-61eb-45bb-9e13-34b7a2519dd5") : secret "samples-operator-tls" not found
Nov 03 11:50:55.951024 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950563    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950557868 +0000 UTC m=+8.963024509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:55.951024 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950579    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls podName:eccf3810-774d-4fe5-9516-6c08440c1505 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950574102 +0000 UTC m=+8.963040742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq6v7" (UID: "eccf3810-774d-4fe5-9516-6c08440c1505") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:55.951024 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950607    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:55.951024 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950619    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:55.951024 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:55.950663    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:50:59.950653097 +0000 UTC m=+8.963119757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:58.315440 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:58.315406    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:58.498389 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:58.498353    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:50:58.558111 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:58.558081    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7bdv2_cb7f0718-4ffa-4b52-abfd-0666d8a6cfb2/dns-node-resolver/0.log"
Nov 03 11:50:59.193529 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.193501    4909 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 11:50:59.758495 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.758468    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcljv_d378c41e-4f5f-4f08-b051-23694fee59d4/node-ca/0.log"
Nov 03 11:50:59.986611 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986577    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:50:59.986806 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986628    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:50:59.986806 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986661    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:59.986806 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986713    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:50:59.986806 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986739    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:59.986806 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.986789    4909 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.986819    4909 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.986841    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle podName:71b27f5e-a0e1-4ba7-a4af-8282a4639d41 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:07.986823931 +0000 UTC m=+16.999290588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle") pod "router-default-b5bc54f87-gkx8h" (UID: "71b27f5e-a0e1-4ba7-a4af-8282a4639d41") : configmap references non-existent config key: service-ca.crt
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.986864    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs podName:dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:07.986849319 +0000 UTC m=+16.999315974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs") pod "network-metrics-daemon-cs8vd" (UID: "dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1") : secret "metrics-daemon-secret" not found
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.986880    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls podName:cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:07.986872378 +0000 UTC m=+16.999339035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xtwgh" (UID: "cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8") : secret "cluster-monitoring-operator-tls" not found
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986898    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986932    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986958    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986982    4909 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987048    4909 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:59.987072 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.986988    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987102    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls podName:4a8522b9-5150-4e60-95bb-f9b04a71add9 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:07.987084512 +0000 UTC m=+16.999551169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls") pod "dns-default-w7l5k" (UID: "4a8522b9-5150-4e60-95bb-f9b04a71add9") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987051    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.987128    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987151    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796696b5b-mlp66: secret "image-registry-tls" not found
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987229    4909 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987244    4909 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9dcd8545-gs4r8: secret "image-registry-tls" not found
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987233    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls podName:188932f7-09f2-4a38-94ec-8db245a5c89a nodeName:}" failed. No retries permitted until 2026-11-03 11:51:07.987216527 +0000 UTC m=+16.999683181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls") pod "image-registry-6796696b5b-mlp66" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a") : secret "image-registry-tls" not found
Nov 03 11:50:59.987623 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:50:59.987295    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls podName:0f923698-d3d2-4118-8f54-eb36c5f29983 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:07.987282274 +0000 UTC m=+16.999748932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls") pod "image-registry-b9dcd8545-gs4r8" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983") : secret "image-registry-tls" not found
Nov 03 11:50:59.989126 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.989105    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eccf3810-774d-4fe5-9516-6c08440c1505-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq6v7\" (UID: \"eccf3810-774d-4fe5-9516-6c08440c1505\") " pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:50:59.989398 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.989376    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f115611-61eb-45bb-9e13-34b7a2519dd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j2dmb\" (UID: \"4f115611-61eb-45bb-9e13-34b7a2519dd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:50:59.989469 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.989394    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-metrics-certs\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:50:59.989469 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:50:59.989398    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be887ffe-4891-4231-93ee-13d03c4efd85-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d2xt7\" (UID: \"be887ffe-4891-4231-93ee-13d03c4efd85\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:51:00.008389 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.008367    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"
Nov 03 11:51:00.009563 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.009528    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"
Nov 03 11:51:00.018858 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.018752    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bq6v7"
Nov 03 11:51:00.473180 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.473150    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-d2xt7"]
Nov 03 11:51:00.511561 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:00.511533    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccf3810_774d_4fe5_9516_6c08440c1505.slice/crio-aa8ca7fa07f7ca736c78d2a21864ea731fc8fe468d79b841e73f0c08a319cc1b WatchSource:0}: Error finding container aa8ca7fa07f7ca736c78d2a21864ea731fc8fe468d79b841e73f0c08a319cc1b: Status 404 returned error can't find the container with id aa8ca7fa07f7ca736c78d2a21864ea731fc8fe468d79b841e73f0c08a319cc1b
Nov 03 11:51:00.524116 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.521596    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb"]
Nov 03 11:51:00.524116 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.521736    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bq6v7"]
Nov 03 11:51:00.525048 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:00.525018    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe887ffe_4891_4231_93ee_13d03c4efd85.slice/crio-4ffb0326d8d3b946cba41fb8759436a1c9cae37e8f5f722ffef6e85bf37b7602 WatchSource:0}: Error finding container 4ffb0326d8d3b946cba41fb8759436a1c9cae37e8f5f722ffef6e85bf37b7602: Status 404 returned error can't find the container with id 4ffb0326d8d3b946cba41fb8759436a1c9cae37e8f5f722ffef6e85bf37b7602
Nov 03 11:51:00.558190 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.558174    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7j2qr_358ed64f-afa7-41b0-9e8f-d981bebf2750/serve-healthcheck-canary/0.log"
Nov 03 11:51:00.758181 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.758106    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-g4hph_85af2d99-dfe2-4630-b5f7-cf96ca0ca5b3/migrator/0.log"
Nov 03 11:51:00.886208 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:00.886179    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qjf7"
Nov 03 11:51:01.158929 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:01.158821    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m_8c1607bc-d5e4-46b1-be5e-e27e7462a967/kube-storage-version-migrator-operator/0.log"
Nov 03 11:51:01.201381 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:01.201348    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq6v7" event={"ID":"eccf3810-774d-4fe5-9516-6c08440c1505","Type":"ContainerStarted","Data":"e8059bffb9945602158f485b50e16ca6e5f20c8aeef227257acaa8f0a0cab06f"}
Nov 03 11:51:01.201562 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:01.201408    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq6v7" event={"ID":"eccf3810-774d-4fe5-9516-6c08440c1505","Type":"ContainerStarted","Data":"aa8ca7fa07f7ca736c78d2a21864ea731fc8fe468d79b841e73f0c08a319cc1b"}
Nov 03 11:51:01.202536 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:01.202513    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7" event={"ID":"be887ffe-4891-4231-93ee-13d03c4efd85","Type":"ContainerStarted","Data":"4ffb0326d8d3b946cba41fb8759436a1c9cae37e8f5f722ffef6e85bf37b7602"}
Nov 03 11:51:01.203572 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:01.203548    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb" event={"ID":"4f115611-61eb-45bb-9e13-34b7a2519dd5","Type":"ContainerStarted","Data":"59a6de0c77f1a88a1249424bec450637434e5d958c7859141746332c4591a8c7"}
Nov 03 11:51:01.526775 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:01.526739    4909 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:51:01.527187 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:01.527166    4909 scope.go:122] "RemoveContainer" containerID="33d6c027dcd7b7108e62bb871f2a46add2683efcb5fafaf5f13b08febbecb0a0"
Nov 03 11:51:01.527742 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:51:01.527379    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rmmzr_openshift-console-operator(70ab9d5e-f98a-4b5a-b72b-175e9182f488)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" podUID="70ab9d5e-f98a-4b5a-b72b-175e9182f488"
Nov 03 11:51:02.209863 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:02.209825    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq6v7" event={"ID":"eccf3810-774d-4fe5-9516-6c08440c1505","Type":"ContainerStarted","Data":"6c7884011135299f456e463cb1bdb83356077503d30451f8920715507f01ca80"}
Nov 03 11:51:02.211555 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:02.211528    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-d2xt7" event={"ID":"be887ffe-4891-4231-93ee-13d03c4efd85","Type":"ContainerStarted","Data":"66fb7e096740f7f895bffd61bbedee825f4a073d04be12d44b1e4b0db850d157"}
Nov 03 11:51:03.116388 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.116358    4909 scope.go:122] "RemoveContainer" containerID="33d6c027dcd7b7108e62bb871f2a46add2683efcb5fafaf5f13b08febbecb0a0"
Nov 03 11:51:03.218721 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.218666    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb" event={"ID":"4f115611-61eb-45bb-9e13-34b7a2519dd5","Type":"ContainerStarted","Data":"25e49cb76ca2bebfea59ee0d5d319c21d22deba18cd64192ff6ffc87c47bc853"}
Nov 03 11:51:03.219172 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.218757    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j2dmb" event={"ID":"4f115611-61eb-45bb-9e13-34b7a2519dd5","Type":"ContainerStarted","Data":"cc2db9cb6aa02c4495ddf84b40f7390179698cfc207c89724155a11a72e65c0c"}
Nov 03 11:51:03.220812 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.220793    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 11:51:03.220937 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.220909    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" event={"ID":"70ab9d5e-f98a-4b5a-b72b-175e9182f488","Type":"ContainerStarted","Data":"beb996fd7d3af7577baeb63357823801f5f9eaa9ada1859ece8bb32045df447d"}
Nov 03 11:51:03.221255 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.221240    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:51:03.222564 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.222543    4909 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rmmzr container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Nov 03 11:51:03.222655 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:03.222578    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr" podUID="70ab9d5e-f98a-4b5a-b72b-175e9182f488" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Nov 03 11:51:04.228959 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:04.228923    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq6v7" event={"ID":"eccf3810-774d-4fe5-9516-6c08440c1505","Type":"ContainerStarted","Data":"cfe777301d59ddac7aa687ed7c4c735477b6e9e92d442d8e28c5689059c23686"}
Nov 03 11:51:04.233084 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:04.233059    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rmmzr"
Nov 03 11:51:08.055799 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.055761    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:51:08.056210 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.055808    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:51:08.056210 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.055832    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:51:08.056210 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.055865    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:51:08.056210 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.055911    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:51:08.056210 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.055929    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:51:08.056786 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.056753    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b27f5e-a0e1-4ba7-a4af-8282a4639d41-service-ca-bundle\") pod \"router-default-b5bc54f87-gkx8h\" (UID: \"71b27f5e-a0e1-4ba7-a4af-8282a4639d41\") " pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:51:08.058556 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.058523    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"image-registry-6796696b5b-mlp66\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") " pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:51:08.058654 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.058619    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a8522b9-5150-4e60-95bb-f9b04a71add9-metrics-tls\") pod \"dns-default-w7l5k\" (UID: \"4a8522b9-5150-4e60-95bb-f9b04a71add9\") " pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:51:08.058713 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.058672    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"image-registry-b9dcd8545-gs4r8\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") " pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:51:08.058943 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.058925    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1-metrics-certs\") pod \"network-metrics-daemon-cs8vd\" (UID: \"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1\") " pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:51:08.059104 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.059087    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xtwgh\" (UID: \"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:51:08.104574 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.104548    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:51:08.107271 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.107253    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:51:08.108440 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.108425    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"
Nov 03 11:51:08.113950 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.113931    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cs8vd"
Nov 03 11:51:08.116148 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.116116    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:51:08.116148 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.116136    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:51:08.586158 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.586080    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7l5k"]
Nov 03 11:51:08.590920 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:08.590673    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8522b9_5150_4e60_95bb_f9b04a71add9.slice/crio-870d827ed5b3421ab6b315a606ceba34957ce4ba45aa01c9d78d103cc2b3b1fc WatchSource:0}: Error finding container 870d827ed5b3421ab6b315a606ceba34957ce4ba45aa01c9d78d103cc2b3b1fc: Status 404 returned error can't find the container with id 870d827ed5b3421ab6b315a606ceba34957ce4ba45aa01c9d78d103cc2b3b1fc
Nov 03 11:51:08.594009 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:08.593983    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb95efe0_e733_48c0_9ff5_a92a0f1ae4e8.slice/crio-f28338366f1fff359d2effaeec95cfb9b1afe7bc3e04bcfb4c2197310fa673e7 WatchSource:0}: Error finding container f28338366f1fff359d2effaeec95cfb9b1afe7bc3e04bcfb4c2197310fa673e7: Status 404 returned error can't find the container with id f28338366f1fff359d2effaeec95cfb9b1afe7bc3e04bcfb4c2197310fa673e7
Nov 03 11:51:08.594750 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:08.594731    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee9a5c2_6d2b_41e4_80b6_8ba17ca79cf1.slice/crio-2bfcc775c02f277ecc0c176663540a723015eeeecb2625524b476da1cbe6bedd WatchSource:0}: Error finding container 2bfcc775c02f277ecc0c176663540a723015eeeecb2625524b476da1cbe6bedd: Status 404 returned error can't find the container with id 2bfcc775c02f277ecc0c176663540a723015eeeecb2625524b476da1cbe6bedd
Nov 03 11:51:08.625565 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:08.625530    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b27f5e_a0e1_4ba7_a4af_8282a4639d41.slice/crio-53645f08ffacdc37ae448fd02c0859f2768c0f2230e99e2c686789f2ee6de276 WatchSource:0}: Error finding container 53645f08ffacdc37ae448fd02c0859f2768c0f2230e99e2c686789f2ee6de276: Status 404 returned error can't find the container with id 53645f08ffacdc37ae448fd02c0859f2768c0f2230e99e2c686789f2ee6de276
Nov 03 11:51:08.626147 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:08.626100    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188932f7_09f2_4a38_94ec_8db245a5c89a.slice/crio-6dceaa971711660e8606419b05b342b2d2c7a66bb7c9729442ce69c6fa62da46 WatchSource:0}: Error finding container 6dceaa971711660e8606419b05b342b2d2c7a66bb7c9729442ce69c6fa62da46: Status 404 returned error can't find the container with id 6dceaa971711660e8606419b05b342b2d2c7a66bb7c9729442ce69c6fa62da46
Nov 03 11:51:08.632097 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.632070    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cs8vd"]
Nov 03 11:51:08.632635 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.632616    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh"]
Nov 03 11:51:08.635528 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.635509    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b5bc54f87-gkx8h"]
Nov 03 11:51:08.638521 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:08.638501    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f923698_d3d2_4118_8f54_eb36c5f29983.slice/crio-e036d25ef3ba621657c738e67a58b9e89651c3cc5d56fbb45c0e29dd2651ea25 WatchSource:0}: Error finding container e036d25ef3ba621657c738e67a58b9e89651c3cc5d56fbb45c0e29dd2651ea25: Status 404 returned error can't find the container with id e036d25ef3ba621657c738e67a58b9e89651c3cc5d56fbb45c0e29dd2651ea25
Nov 03 11:51:08.678931 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.678906    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6796696b5b-mlp66"]
Nov 03 11:51:08.679575 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:08.679555    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9dcd8545-gs4r8"]
Nov 03 11:51:09.247157 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.247123    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b5bc54f87-gkx8h" event={"ID":"71b27f5e-a0e1-4ba7-a4af-8282a4639d41","Type":"ContainerStarted","Data":"32f0e71a2c1098b34decfb29d24e700f555f56fa9bd2faf21bb03165d3ade229"}
Nov 03 11:51:09.247606 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.247171    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b5bc54f87-gkx8h" event={"ID":"71b27f5e-a0e1-4ba7-a4af-8282a4639d41","Type":"ContainerStarted","Data":"53645f08ffacdc37ae448fd02c0859f2768c0f2230e99e2c686789f2ee6de276"}
Nov 03 11:51:09.248404 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.248383    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7l5k" event={"ID":"4a8522b9-5150-4e60-95bb-f9b04a71add9","Type":"ContainerStarted","Data":"870d827ed5b3421ab6b315a606ceba34957ce4ba45aa01c9d78d103cc2b3b1fc"}
Nov 03 11:51:09.249567 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.249541    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh" event={"ID":"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8","Type":"ContainerStarted","Data":"f28338366f1fff359d2effaeec95cfb9b1afe7bc3e04bcfb4c2197310fa673e7"}
Nov 03 11:51:09.250682 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.250660    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cs8vd" event={"ID":"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1","Type":"ContainerStarted","Data":"2bfcc775c02f277ecc0c176663540a723015eeeecb2625524b476da1cbe6bedd"}
Nov 03 11:51:09.252277 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.252249    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8" event={"ID":"0f923698-d3d2-4118-8f54-eb36c5f29983","Type":"ContainerStarted","Data":"0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3"}
Nov 03 11:51:09.252392 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.252293    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8" event={"ID":"0f923698-d3d2-4118-8f54-eb36c5f29983","Type":"ContainerStarted","Data":"e036d25ef3ba621657c738e67a58b9e89651c3cc5d56fbb45c0e29dd2651ea25"}
Nov 03 11:51:09.252392 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.252360    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:51:09.253570 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.253551    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796696b5b-mlp66" event={"ID":"188932f7-09f2-4a38-94ec-8db245a5c89a","Type":"ContainerStarted","Data":"200737f1893ba833713da76ed23dbad3332527c45e0c89dc768073f0b52b8c76"}
Nov 03 11:51:09.253650 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.253582    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796696b5b-mlp66" event={"ID":"188932f7-09f2-4a38-94ec-8db245a5c89a","Type":"ContainerStarted","Data":"6dceaa971711660e8606419b05b342b2d2c7a66bb7c9729442ce69c6fa62da46"}
Nov 03 11:51:09.253689 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:09.253679    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:51:09.564674 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:51:09.564599    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:51:10.105531 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:10.105455    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:51:10.108873 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:10.108580    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:51:10.259981 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:10.259945    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:51:10.260575 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:10.260552    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b5bc54f87-gkx8h"
Nov 03 11:51:11.263990 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:11.263959    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7l5k" event={"ID":"4a8522b9-5150-4e60-95bb-f9b04a71add9","Type":"ContainerStarted","Data":"a7302443337dc1686d13486ea2a66c76c46e0ae194bc62754177e0b79a7ab1e5"}
Nov 03 11:51:11.264453 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:11.264007    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7l5k" event={"ID":"4a8522b9-5150-4e60-95bb-f9b04a71add9","Type":"ContainerStarted","Data":"b62cdfb5428da6989c0f7c4282d7c9898bcb0aa3989108856d7f3a9b6fb6f72e"}
Nov 03 11:51:11.264453 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:11.264043    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:51:11.265748 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:11.265723    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xtwgh" event={"ID":"cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8","Type":"ContainerStarted","Data":"57d4c29a0af4c67e3cb5c71e1dd7e13b754ef2473a09d7dff9fb84bf8191903a"}
Nov 03 11:51:11.267622 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:11.267595    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cs8vd" event={"ID":"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1","Type":"ContainerStarted","Data":"d27f2a3ddcde5c0b28588fb473e2d6a720a22e89d6c560c7c030e02dea5f8a7b"}
Nov 03 11:51:11.267883 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:11.267640    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cs8vd" event={"ID":"dee9a5c2-6d2b-41e4-80b6-8ba17ca79cf1","Type":"ContainerStarted","Data":"cad7ff70aa66ce3e53bf2e2d6a9339b8bdf926f564d1df4a67888734fd964ff3"}
Nov 03 11:51:12.159205 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:12.159172    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6796696b5b-mlp66"]
Nov 03 11:51:17.261720 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:17.261675    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-df7t5"
Nov 03 11:51:21.272932 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:21.272899    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w7l5k"
Nov 03 11:51:23.031402 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.031370    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dzlns"]
Nov 03 11:51:23.081557 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.081524    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171052 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171016    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0834d68-fe3a-4284-868d-04034a3704b3-metrics-client-ca\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171220 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171070    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171220 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171099    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-sys\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171220 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171195    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72w5g\" (UniqueName: \"kubernetes.io/projected/d0834d68-fe3a-4284-868d-04034a3704b3-kube-api-access-72w5g\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171356 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171245    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-wtmp\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171356 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171288    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171356 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171321    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-tls\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171458 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171374    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-root\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.171458 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.171395    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-textfile\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.192566 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.192531    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 11:51:23.192733 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.192585    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 11:51:23.238559 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.238532    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s8zh8\""
Nov 03 11:51:23.239193 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.239161    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 11:51:23.239285 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.239243    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 11:51:23.272057 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272021    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-tls\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272175 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272063    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-root\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272175 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272091    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-textfile\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272175 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272127    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0834d68-fe3a-4284-868d-04034a3704b3-metrics-client-ca\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272175 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272145    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-root\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272175 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272169    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272435 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272195    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-sys\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272435 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272224    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72w5g\" (UniqueName: \"kubernetes.io/projected/d0834d68-fe3a-4284-868d-04034a3704b3-kube-api-access-72w5g\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272435 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272257    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-wtmp\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272435 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272281    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272435 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272310    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-sys\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272677 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272447    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-wtmp\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272677 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272560    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-textfile\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272927 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272905    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0834d68-fe3a-4284-868d-04034a3704b3-metrics-client-ca\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.272998 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.272908    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.274495 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.274474    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-tls\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.274584 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.274516    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0834d68-fe3a-4284-868d-04034a3704b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.326974 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.326909    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72w5g\" (UniqueName: \"kubernetes.io/projected/d0834d68-fe3a-4284-868d-04034a3704b3-kube-api-access-72w5g\") pod \"node-exporter-dzlns\" (UID: \"d0834d68-fe3a-4284-868d-04034a3704b3\") " pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.432050 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:23.432022    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dzlns"
Nov 03 11:51:23.476500 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:51:23.476464    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0834d68_fe3a_4284_868d_04034a3704b3.slice/crio-fd4caf365690991e452af0e50328c0d20746182f87e2aa22bf1ee868f9a6f57f WatchSource:0}: Error finding container fd4caf365690991e452af0e50328c0d20746182f87e2aa22bf1ee868f9a6f57f: Status 404 returned error can't find the container with id fd4caf365690991e452af0e50328c0d20746182f87e2aa22bf1ee868f9a6f57f
Nov 03 11:51:24.315528 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:24.315492    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dzlns" event={"ID":"d0834d68-fe3a-4284-868d-04034a3704b3","Type":"ContainerStarted","Data":"fd4caf365690991e452af0e50328c0d20746182f87e2aa22bf1ee868f9a6f57f"}
Nov 03 11:51:25.542225 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:25.542204    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gkt2k"
Nov 03 11:51:26.324364 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:26.324329    4909 generic.go:356] "Generic (PLEG): container finished" podID="d0834d68-fe3a-4284-868d-04034a3704b3" containerID="daf29cec238b7ea0815b69f02533c0fb572c0e8afb3fb066f9f998aec29add35" exitCode=0
Nov 03 11:51:26.324566 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:26.324417    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dzlns" event={"ID":"d0834d68-fe3a-4284-868d-04034a3704b3","Type":"ContainerDied","Data":"daf29cec238b7ea0815b69f02533c0fb572c0e8afb3fb066f9f998aec29add35"}
Nov 03 11:51:27.335174 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:27.335140    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dzlns" event={"ID":"d0834d68-fe3a-4284-868d-04034a3704b3","Type":"ContainerStarted","Data":"d6f1b7c3e11b4c453ed02259d6b945bfd767d6a13946716faad6f89aaa16d7aa"}
Nov 03 11:51:27.335568 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:27.335229    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dzlns" event={"ID":"d0834d68-fe3a-4284-868d-04034a3704b3","Type":"ContainerStarted","Data":"664105ca21b0e86c3fff3e02584ddd268f65b0e9aa6db4e6915941f20f2730d5"}
Nov 03 11:51:27.443082 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:27.443018    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dzlns" podStartSLOduration=3.409910505 podStartE2EDuration="5.442999976s" totalImagesPullingTime="1.954134782s" totalInitContainerRuntime="78.954689ms" isStatefulPod=true podCreationTimestamp="2026-11-03 11:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:27.441924973 +0000 UTC m=+36.454391635" watchObservedRunningTime="2026-11-03 11:51:27.442999976 +0000 UTC m=+36.455466641"
Nov 03 11:51:28.119955 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:28.119915    4909 patch_prober.go:29] interesting pod/image-registry-b9dcd8545-gs4r8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 11:51:28.120109 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:28.119969    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8" podUID="0f923698-d3d2-4118-8f54-eb36c5f29983" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:51:30.263425 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:30.263399    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:51:32.165015 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:32.164943    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:51:34.845848 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:34.845812    4909 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-df7t5_77d106d7-59a7-459c-b012-aaa087cb50d9/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:35.846622 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:35.846592    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9dcd8545-gs4r8"]
Nov 03 11:51:37.180393 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.180335    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6796696b5b-mlp66" podUID="188932f7-09f2-4a38-94ec-8db245a5c89a" containerName="registry" containerID="cri-o://200737f1893ba833713da76ed23dbad3332527c45e0c89dc768073f0b52b8c76" gracePeriod=30
Nov 03 11:51:37.381305 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.381271    4909 generic.go:356] "Generic (PLEG): container finished" podID="188932f7-09f2-4a38-94ec-8db245a5c89a" containerID="200737f1893ba833713da76ed23dbad3332527c45e0c89dc768073f0b52b8c76" exitCode=0
Nov 03 11:51:37.381465 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.381335    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796696b5b-mlp66" event={"ID":"188932f7-09f2-4a38-94ec-8db245a5c89a","Type":"ContainerDied","Data":"200737f1893ba833713da76ed23dbad3332527c45e0c89dc768073f0b52b8c76"}
Nov 03 11:51:37.425221 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.425199    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:51:37.491175 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491147    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491286 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491183    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491286 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491211    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491286 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491253    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491286 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491284    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491505 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491324    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491505 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491358    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491505 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491396    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\") pod \"188932f7-09f2-4a38-94ec-8db245a5c89a\" (UID: \"188932f7-09f2-4a38-94ec-8db245a5c89a\") "
Nov 03 11:51:37.491775 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491746    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:51:37.491939 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.491894    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:51:37.494164 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.494130    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:51:37.494263 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.494175    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:51:37.494328 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.494289    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:51:37.494419 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.494368    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:51:37.494471 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.494448    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "kube-api-access-g8hsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:51:37.498565 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.498546    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted" pod "188932f7-09f2-4a38-94ec-8db245a5c89a" (UID: "188932f7-09f2-4a38-94ec-8db245a5c89a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:51:37.592738 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592708    4909 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-installation-pull-secrets\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:37.592738 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592738    4909 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-bound-sa-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:37.592900 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592748    4909 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-certificates\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:37.592900 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592758    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hsb\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-kube-api-access-g8hsb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:37.592900 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592767    4909 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/188932f7-09f2-4a38-94ec-8db245a5c89a-image-registry-private-configuration\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:37.592900 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592776    4909 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/188932f7-09f2-4a38-94ec-8db245a5c89a-ca-trust-extracted\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:37.592900 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592785    4909 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/188932f7-09f2-4a38-94ec-8db245a5c89a-registry-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:37.592900 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:37.592794    4909 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/188932f7-09f2-4a38-94ec-8db245a5c89a-trusted-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:51:38.386184 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:38.386156    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796696b5b-mlp66"
Nov 03 11:51:38.386582 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:38.386156    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796696b5b-mlp66" event={"ID":"188932f7-09f2-4a38-94ec-8db245a5c89a","Type":"ContainerDied","Data":"6dceaa971711660e8606419b05b342b2d2c7a66bb7c9729442ce69c6fa62da46"}
Nov 03 11:51:38.386582 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:38.386293    4909 scope.go:122] "RemoveContainer" containerID="200737f1893ba833713da76ed23dbad3332527c45e0c89dc768073f0b52b8c76"
Nov 03 11:51:38.584264 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:38.584231    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6796696b5b-mlp66"]
Nov 03 11:51:38.589004 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:38.588976    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6796696b5b-mlp66"]
Nov 03 11:51:39.121221 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:39.121189    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188932f7-09f2-4a38-94ec-8db245a5c89a" path="/var/lib/kubelet/pods/188932f7-09f2-4a38-94ec-8db245a5c89a/volumes"
Nov 03 11:51:39.562723 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:51:39.562666    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:51:57.453834 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:57.453807    4909 generic.go:356] "Generic (PLEG): container finished" podID="8c1607bc-d5e4-46b1-be5e-e27e7462a967" containerID="cbb9c1644f40746f2056286aac789c89044f86e5257b41f051b4fac467511d2a" exitCode=0
Nov 03 11:51:57.454169 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:57.453878    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m" event={"ID":"8c1607bc-d5e4-46b1-be5e-e27e7462a967","Type":"ContainerDied","Data":"cbb9c1644f40746f2056286aac789c89044f86e5257b41f051b4fac467511d2a"}
Nov 03 11:51:57.454169 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:57.454137    4909 scope.go:122] "RemoveContainer" containerID="cbb9c1644f40746f2056286aac789c89044f86e5257b41f051b4fac467511d2a"
Nov 03 11:51:58.459016 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:51:58.458984    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7jl7m" event={"ID":"8c1607bc-d5e4-46b1-be5e-e27e7462a967","Type":"ContainerStarted","Data":"39217d9e0051ce64b697fffe7b71fec68abb2559e8e8770c99a40026b6ef4d39"}
Nov 03 11:52:00.865862 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:00.865823    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8" podUID="0f923698-d3d2-4118-8f54-eb36c5f29983" containerName="registry" containerID="cri-o://0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3" gracePeriod=30
Nov 03 11:52:01.117767 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.117710    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:52:01.165526 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.165498    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.165643 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.165538    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.165643 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.165562    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.165643 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.165605    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.165643 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.165641    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.165873 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.165674    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.165873 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.165734    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.166268 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.166174    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\") pod \"0f923698-d3d2-4118-8f54-eb36c5f29983\" (UID: \"0f923698-d3d2-4118-8f54-eb36c5f29983\") "
Nov 03 11:52:01.166487 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.166170    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:01.166623 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.166601    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:01.168410 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.168375    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:01.168513 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.168468    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:01.168513 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.168471    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "kube-api-access-ncm5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:01.168513 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.168493    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:01.168617 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.168521    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:01.175112 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.175066    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted" pod "0f923698-d3d2-4118-8f54-eb36c5f29983" (UID: "0f923698-d3d2-4118-8f54-eb36c5f29983"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:01.267620 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267593    4909 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-certificates\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.267620 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267616    4909 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-bound-sa-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.267620 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267627    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncm5f\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-kube-api-access-ncm5f\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.267832 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267636    4909 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f923698-d3d2-4118-8f54-eb36c5f29983-trusted-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.267832 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267645    4909 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f923698-d3d2-4118-8f54-eb36c5f29983-ca-trust-extracted\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.267832 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267654    4909 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f923698-d3d2-4118-8f54-eb36c5f29983-registry-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.267832 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267662    4909 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-image-registry-private-configuration\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.267832 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.267671    4909 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f923698-d3d2-4118-8f54-eb36c5f29983-installation-pull-secrets\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.470199 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.470164    4909 generic.go:356] "Generic (PLEG): container finished" podID="0f923698-d3d2-4118-8f54-eb36c5f29983" containerID="0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3" exitCode=0
Nov 03 11:52:01.470348 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.470205    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8" event={"ID":"0f923698-d3d2-4118-8f54-eb36c5f29983","Type":"ContainerDied","Data":"0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3"}
Nov 03 11:52:01.470348 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.470220    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8"
Nov 03 11:52:01.470348 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.470247    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9dcd8545-gs4r8" event={"ID":"0f923698-d3d2-4118-8f54-eb36c5f29983","Type":"ContainerDied","Data":"e036d25ef3ba621657c738e67a58b9e89651c3cc5d56fbb45c0e29dd2651ea25"}
Nov 03 11:52:01.470348 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.470269    4909 scope.go:122] "RemoveContainer" containerID="0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3"
Nov 03 11:52:01.480740 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.480714    4909 scope.go:122] "RemoveContainer" containerID="0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3"
Nov 03 11:52:01.481838 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:52:01.481805    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3\": container with ID starting with 0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3 not found: ID does not exist" containerID="0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3"
Nov 03 11:52:01.481946 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.481847    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3"} err="failed to get container status \"0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3\": rpc error: code = NotFound desc = could not find container \"0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3\": container with ID starting with 0a2a391e9879eb4f756a99cd7a628c03d9a6bde812ced8874958b5d3480013c3 not found: ID does not exist"
Nov 03 11:52:01.622813 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.622772    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9dcd8545-gs4r8"]
Nov 03 11:52:01.626494 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:01.626470    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b9dcd8545-gs4r8"]
Nov 03 11:52:03.121304 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:03.121273    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f923698-d3d2-4118-8f54-eb36c5f29983" path="/var/lib/kubelet/pods/0f923698-d3d2-4118-8f54-eb36c5f29983/volumes"
Nov 03 11:52:07.491442 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:07.491404    4909 generic.go:356] "Generic (PLEG): container finished" podID="bab06cb5-f005-4c43-a247-5f38134286cb" containerID="e0ac9de693deae2341e2c3fe7e854ecc2d191944f775e48fc7b0359cdc7076f5" exitCode=0
Nov 03 11:52:07.491856 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:07.491458    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t" event={"ID":"bab06cb5-f005-4c43-a247-5f38134286cb","Type":"ContainerDied","Data":"e0ac9de693deae2341e2c3fe7e854ecc2d191944f775e48fc7b0359cdc7076f5"}
Nov 03 11:52:07.491856 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:07.491735    4909 scope.go:122] "RemoveContainer" containerID="e0ac9de693deae2341e2c3fe7e854ecc2d191944f775e48fc7b0359cdc7076f5"
Nov 03 11:52:08.496538 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:08.496504    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5t86t" event={"ID":"bab06cb5-f005-4c43-a247-5f38134286cb","Type":"ContainerStarted","Data":"4a295814c5d4ad1a9fa769c89cf251b57931b63340e7aa422bac8f22d440d34b"}
Nov 03 11:52:09.562329 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:52:09.562294    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:52:12.511758 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:12.511728    4909 generic.go:356] "Generic (PLEG): container finished" podID="ef83543a-8e80-486b-adeb-4d5aad83d4b0" containerID="59902d0ab67d6106e17e92018899165a7e695e415605d4352af1604975de2c2e" exitCode=0
Nov 03 11:52:12.512156 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:12.511809    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8cwrc" event={"ID":"ef83543a-8e80-486b-adeb-4d5aad83d4b0","Type":"ContainerDied","Data":"59902d0ab67d6106e17e92018899165a7e695e415605d4352af1604975de2c2e"}
Nov 03 11:52:12.512227 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:12.512159    4909 scope.go:122] "RemoveContainer" containerID="59902d0ab67d6106e17e92018899165a7e695e415605d4352af1604975de2c2e"
Nov 03 11:52:13.516628 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:52:13.516594    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8cwrc" event={"ID":"ef83543a-8e80-486b-adeb-4d5aad83d4b0","Type":"ContainerStarted","Data":"fd5ccf117207c0c757f1ddd65b96d2534f6fc3975c836bb2627bad69bbc555e6"}
Nov 03 11:52:39.561856 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:52:39.561822    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:53:09.563256 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:53:09.563174    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:53:39.562770 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:53:39.562731    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:54:09.562600 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:54:09.562562    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:54:39.561889 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:54:39.561816    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:55:09.562819 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:55:09.562783    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:55:37.797719 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.797665    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"]
Nov 03 11:55:37.798194 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.798015    4909 cpu_manager.go:414] "containerMap: removing container" podUID="0f923698-d3d2-4118-8f54-eb36c5f29983" containerName="registry"
Nov 03 11:55:37.798194 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.798029    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f923698-d3d2-4118-8f54-eb36c5f29983" containerName="registry"
Nov 03 11:55:37.798194 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.798054    4909 cpu_manager.go:414] "containerMap: removing container" podUID="188932f7-09f2-4a38-94ec-8db245a5c89a" containerName="registry"
Nov 03 11:55:37.798194 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.798059    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188932f7-09f2-4a38-94ec-8db245a5c89a" containerName="registry"
Nov 03 11:55:37.798194 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.798112    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f923698-d3d2-4118-8f54-eb36c5f29983" containerName="registry"
Nov 03 11:55:37.798194 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.798122    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="188932f7-09f2-4a38-94ec-8db245a5c89a" containerName="registry"
Nov 03 11:55:37.801018 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.801001    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.839624 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.839599    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.839746 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.839637    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.839746 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.839674    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.839842 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.839754    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzwh\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-kube-api-access-wlzwh\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.839842 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.839819    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.839930 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.839860    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.839969 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.839935    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab29b377-b610-482b-a8c0-817fdc612fc0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.851944 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.851919    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 11:55:37.852049 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.851942    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 11:55:37.852049 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.851960    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 11:55:37.852049 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.851986    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 11:55:37.852151 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.852090    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rhbbs\""
Nov 03 11:55:37.860887 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.860863    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"]
Nov 03 11:55:37.940258 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.940231    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.940369 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.940264    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzwh\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-kube-api-access-wlzwh\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.940369 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.940287    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.940481 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.940416    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.940537 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.940483    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab29b377-b610-482b-a8c0-817fdc612fc0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.940537 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.940519    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.940664 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.940546    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.941057 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.941025    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.942808 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.942784    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab29b377-b610-482b-a8c0-817fdc612fc0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.942808 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.942799    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.942943 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.942896    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.942980 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.942959    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.992920 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.992899    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:37.993582 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:37.993561    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzwh\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-kube-api-access-wlzwh\") pod \"istiod-openshift-gateway-7876bfcb74-4hn7s\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:38.112824 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:38.112773    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:38.519638 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:38.519610    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"]
Nov 03 11:55:38.522833 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:55:38.522805    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab29b377_b610_482b_a8c0_817fdc612fc0.slice/crio-8f00ab9dc1db179cb9fbb20be2b9c14cbc4e872335efe9834427e36ee17fb0cf WatchSource:0}: Error finding container 8f00ab9dc1db179cb9fbb20be2b9c14cbc4e872335efe9834427e36ee17fb0cf: Status 404 returned error can't find the container with id 8f00ab9dc1db179cb9fbb20be2b9c14cbc4e872335efe9834427e36ee17fb0cf
Nov 03 11:55:39.222710 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:39.222671    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s" event={"ID":"ab29b377-b610-482b-a8c0-817fdc612fc0","Type":"ContainerStarted","Data":"8f00ab9dc1db179cb9fbb20be2b9c14cbc4e872335efe9834427e36ee17fb0cf"}
Nov 03 11:55:39.563178 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:55:39.563103    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:55:43.004134 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:43.004101    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 11:55:43.004442 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:43.004164    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 11:55:43.241733 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:43.241679    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s" event={"ID":"ab29b377-b610-482b-a8c0-817fdc612fc0","Type":"ContainerStarted","Data":"2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9"}
Nov 03 11:55:43.241902 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:43.241826    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:43.443198 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:43.443148    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s" podStartSLOduration=1.964370605 podStartE2EDuration="6.443135918s" totalImagesPullingTime="4.478765313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:55:43.396993461 +0000 UTC m=+292.409460124" watchObservedRunningTime="2026-11-03 11:55:43.443135918 +0000 UTC m=+292.455602572"
Nov 03 11:55:44.247441 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:44.247411    4909 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4hn7s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:55:44.247823 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:44.247465    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s" podUID="ab29b377-b610-482b-a8c0-817fdc612fc0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:55:47.247214 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:47.247184    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:55:51.130892 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:51.130863    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 11:55:51.133067 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:55:51.133047    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 11:56:09.562458 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:56:09.562376    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:56:39.562147 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:56:39.562110    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:56:44.694680 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.694649    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"]
Nov 03 11:56:44.698008 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.697990    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.719824 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.719796    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.719929 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.719834    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.719975 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.719936    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.719975 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.719967    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.720073 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.720061    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.720107 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.720085    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9l8\" (UniqueName: \"kubernetes.io/projected/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-kube-api-access-kl9l8\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.820394 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.820368    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.820540 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.820398    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9l8\" (UniqueName: \"kubernetes.io/projected/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-kube-api-access-kl9l8\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.820540 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.820417    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.820540 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.820437    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.820540 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.820482    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.820540 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.820513    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.820971 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.820950    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.822870 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.822848    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.863674 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.863648    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 11:56:44.863783 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.863723    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 11:56:44.863783 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.863724    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 11:56:44.863783 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.863767    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 11:56:44.863898 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.863823    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 11:56:44.863898 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.863875    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zcjpl\""
Nov 03 11:56:44.873485 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.873463    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.873589 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.873518    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.873589 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.873533    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.874260 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.874237    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9l8\" (UniqueName: \"kubernetes.io/projected/46f281a1-4a78-4ab4-a24a-ae742d01cf7e-kube-api-access-kl9l8\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-m9tt6\" (UID: \"46f281a1-4a78-4ab4-a24a-ae742d01cf7e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:44.928993 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:44.928968    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"]
Nov 03 11:56:45.013590 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:45.013532    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:45.482219 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:45.482186    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"]
Nov 03 11:56:45.484690 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:56:45.484663    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f281a1_4a78_4ab4_a24a_ae742d01cf7e.slice/crio-122ff571969b56df427ab76b30c0f0bcb1fdbf5008d6df966563f7990e59c9f6 WatchSource:0}: Error finding container 122ff571969b56df427ab76b30c0f0bcb1fdbf5008d6df966563f7990e59c9f6: Status 404 returned error can't find the container with id 122ff571969b56df427ab76b30c0f0bcb1fdbf5008d6df966563f7990e59c9f6
Nov 03 11:56:45.486233 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:45.486216    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:56:46.462688 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:46.462657    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6" event={"ID":"46f281a1-4a78-4ab4-a24a-ae742d01cf7e","Type":"ContainerStarted","Data":"122ff571969b56df427ab76b30c0f0bcb1fdbf5008d6df966563f7990e59c9f6"}
Nov 03 11:56:52.489580 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:52.489544    4909 generic.go:356] "Generic (PLEG): container finished" podID="46f281a1-4a78-4ab4-a24a-ae742d01cf7e" containerID="7fa0057e498910b55a3f83df5d26b993c3af7b5c96df6a9fa5a1408e043b9ace" exitCode=0
Nov 03 11:56:52.489936 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:52.489636    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6" event={"ID":"46f281a1-4a78-4ab4-a24a-ae742d01cf7e","Type":"ContainerDied","Data":"7fa0057e498910b55a3f83df5d26b993c3af7b5c96df6a9fa5a1408e043b9ace"}
Nov 03 11:56:53.495342 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:53.495307    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6" event={"ID":"46f281a1-4a78-4ab4-a24a-ae742d01cf7e","Type":"ContainerStarted","Data":"ac258c3bb6ce900d358241154204e4b760bae74db32451634cfa6fe3d000a7c8"}
Nov 03 11:56:53.495724 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:53.495419    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:56:53.619827 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:56:53.619713    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6" podStartSLOduration=1.983180376 podStartE2EDuration="9.619686181s" totalImagesPullingTime="7.17709803s" totalInitContainerRuntime="459.407775ms" isStatefulPod=false podCreationTimestamp="2026-11-03 11:56:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:53.618019147 +0000 UTC m=+362.630485832" watchObservedRunningTime="2026-11-03 11:56:53.619686181 +0000 UTC m=+362.632152846"
Nov 03 11:57:04.502489 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:04.502454    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-m9tt6"
Nov 03 11:57:09.563063 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:57:09.563032    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:57:21.375747 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.375716    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"]
Nov 03 11:57:21.382535 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.382512    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.461031 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.461000    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"]
Nov 03 11:57:21.512612 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.512571    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-tls-certs\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.512789 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.512686    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-tmp\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.512789 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.512747    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrtq\" (UniqueName: \"kubernetes.io/projected/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-kube-api-access-szrtq\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.536157 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.536129    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t7b7z\""
Nov 03 11:57:21.536157 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.536129    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 11:57:21.536313 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.536133    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 11:57:21.613476 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.613444    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-tmp\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.613630 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.613490    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrtq\" (UniqueName: \"kubernetes.io/projected/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-kube-api-access-szrtq\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.613630 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.613530    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-tls-certs\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.615754 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.615724    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-tmp\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.615919 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.615902    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-tls-certs\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.666107 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.666048    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrtq\" (UniqueName: \"kubernetes.io/projected/93aba9c3-9358-4bc6-bd3b-69d1a48bb631-kube-api-access-szrtq\") pod \"kube-auth-proxy-758d88fbcf-b96gm\" (UID: \"93aba9c3-9358-4bc6-bd3b-69d1a48bb631\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:21.694440 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:21.694402    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"
Nov 03 11:57:22.086454 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:22.086429    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm"]
Nov 03 11:57:22.087676 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:57:22.087646    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aba9c3_9358_4bc6_bd3b_69d1a48bb631.slice/crio-dcf798cc693cc27cc7a1c3e45078464a82d8462580e5ec65aabadc0f0ed5d3e2 WatchSource:0}: Error finding container dcf798cc693cc27cc7a1c3e45078464a82d8462580e5ec65aabadc0f0ed5d3e2: Status 404 returned error can't find the container with id dcf798cc693cc27cc7a1c3e45078464a82d8462580e5ec65aabadc0f0ed5d3e2
Nov 03 11:57:22.606712 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:22.606664    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm" event={"ID":"93aba9c3-9358-4bc6-bd3b-69d1a48bb631","Type":"ContainerStarted","Data":"dcf798cc693cc27cc7a1c3e45078464a82d8462580e5ec65aabadc0f0ed5d3e2"}
Nov 03 11:57:27.629155 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:27.629070    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm" event={"ID":"93aba9c3-9358-4bc6-bd3b-69d1a48bb631","Type":"ContainerStarted","Data":"aa04563d58f0c1fc2ba65b29d5f222d39432e4f727ddc109613138853bf18aa2"}
Nov 03 11:57:27.736890 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:27.736845    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-b96gm" podStartSLOduration=1.5306571199999999 podStartE2EDuration="6.736831927s" totalImagesPullingTime="5.206174807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:27.736118506 +0000 UTC m=+396.748585170" watchObservedRunningTime="2026-11-03 11:57:27.736831927 +0000 UTC m=+396.749298618"
Nov 03 11:57:39.562874 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:57:39.562799    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:57:53.206267 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.206230    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 11:57:53.212528 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.212501    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.255713 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.255666    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ba0dd0de-32a0-4a53-a8ab-3c8804f7af66-config-file\") pod \"limitador-limitador-7c9b87d574-xj776\" (UID: \"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.255854 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.255725    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2j4n\" (UniqueName: \"kubernetes.io/projected/ba0dd0de-32a0-4a53-a8ab-3c8804f7af66-kube-api-access-p2j4n\") pod \"limitador-limitador-7c9b87d574-xj776\" (UID: \"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.261337 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.261318    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 11:57:53.307179 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.307159    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 11:57:53.307302 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.307239    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 11:57:53.307302 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.307256    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8xttl\""
Nov 03 11:57:53.310256 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.310229    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 11:57:53.356538 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.356514    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ba0dd0de-32a0-4a53-a8ab-3c8804f7af66-config-file\") pod \"limitador-limitador-7c9b87d574-xj776\" (UID: \"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.356642 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.356545    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j4n\" (UniqueName: \"kubernetes.io/projected/ba0dd0de-32a0-4a53-a8ab-3c8804f7af66-kube-api-access-p2j4n\") pod \"limitador-limitador-7c9b87d574-xj776\" (UID: \"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.357163 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.357138    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ba0dd0de-32a0-4a53-a8ab-3c8804f7af66-config-file\") pod \"limitador-limitador-7c9b87d574-xj776\" (UID: \"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.408862 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.408838    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2j4n\" (UniqueName: \"kubernetes.io/projected/ba0dd0de-32a0-4a53-a8ab-3c8804f7af66-kube-api-access-p2j4n\") pod \"limitador-limitador-7c9b87d574-xj776\" (UID: \"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.499469 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.499404    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 11:57:53.524281 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.524252    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:53.882449 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:53.882264    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 11:57:53.885088 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:57:53.885060    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0dd0de_32a0_4a53_a8ab_3c8804f7af66.slice/crio-3bdeeb65293d74f59c1be93cb4f5bf695a3b4672a6082c3038c5566e8cf8855a WatchSource:0}: Error finding container 3bdeeb65293d74f59c1be93cb4f5bf695a3b4672a6082c3038c5566e8cf8855a: Status 404 returned error can't find the container with id 3bdeeb65293d74f59c1be93cb4f5bf695a3b4672a6082c3038c5566e8cf8855a
Nov 03 11:57:54.733058 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:54.733025    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776" event={"ID":"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66","Type":"ContainerStarted","Data":"3bdeeb65293d74f59c1be93cb4f5bf695a3b4672a6082c3038c5566e8cf8855a"}
Nov 03 11:57:58.752954 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:58.752918    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776" event={"ID":"ba0dd0de-32a0-4a53-a8ab-3c8804f7af66","Type":"ContainerStarted","Data":"21d80dbd7b21bf0fa5979a7c2d7df912c86f6d79a74faffeae109e461be6ce17"}
Nov 03 11:57:58.753351 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:58.753054    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:57:58.861839 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:57:58.861790    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776" podStartSLOduration=1.903387108 podStartE2EDuration="5.861765806s" totalImagesPullingTime="3.958378698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:58.86073014 +0000 UTC m=+427.873196804" watchObservedRunningTime="2026-11-03 11:57:58.861765806 +0000 UTC m=+427.874232466"
Nov 03 11:58:09.563078 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:58:09.563038    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:58:09.758095 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:09.758061    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xj776"
Nov 03 11:58:39.562433 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:58:39.562399    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:58:49.691037 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.691007    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"]
Nov 03 11:58:49.694211 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.694190    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.788683 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.788654    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.788801 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.788711    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.788801 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.788782    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.788891 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.788810    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxk7c\" (UniqueName: \"kubernetes.io/projected/b3f9e19a-8336-4fe1-8ac3-721dcf430675-kube-api-access-zxk7c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.788891 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.788833    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.789000 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.788961    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f9e19a-8336-4fe1-8ac3-721dcf430675-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.789078 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.789061    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.795196 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.795176    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"]
Nov 03 11:58:49.889432 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.889400    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.889585 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.889450    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f9e19a-8336-4fe1-8ac3-721dcf430675-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.889585 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.889512    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.889585 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.889546    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.889585 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.889574    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.889843 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.889599    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.889843 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.889623    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxk7c\" (UniqueName: \"kubernetes.io/projected/b3f9e19a-8336-4fe1-8ac3-721dcf430675-kube-api-access-zxk7c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.890287 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.890261    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.892396 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.892369    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.892486 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.892449    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f9e19a-8336-4fe1-8ac3-721dcf430675-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.892578 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.892562    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.892648 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.892626    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.943115 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.943058    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f9e19a-8336-4fe1-8ac3-721dcf430675-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:49.943471 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:49.943453    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxk7c\" (UniqueName: \"kubernetes.io/projected/b3f9e19a-8336-4fe1-8ac3-721dcf430675-kube-api-access-zxk7c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-klwdt\" (UID: \"b3f9e19a-8336-4fe1-8ac3-721dcf430675\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:50.005330 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:50.005279    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:50.413353 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:50.413212    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"]
Nov 03 11:58:50.416016 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:58:50.415989    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f9e19a_8336_4fe1_8ac3_721dcf430675.slice/crio-194ffd51bca9263e6544764e795c037034c1008acddde42f747431c26271ce16 WatchSource:0}: Error finding container 194ffd51bca9263e6544764e795c037034c1008acddde42f747431c26271ce16: Status 404 returned error can't find the container with id 194ffd51bca9263e6544764e795c037034c1008acddde42f747431c26271ce16
Nov 03 11:58:50.418203 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:50.418172    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 11:58:50.418291 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:50.418233    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 11:58:50.959718 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:50.959670    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt" event={"ID":"b3f9e19a-8336-4fe1-8ac3-721dcf430675","Type":"ContainerStarted","Data":"90a882b90abf07eeda55cb6fa7584398a4a7ddc1e22abcf1913fd5cf4520086e"}
Nov 03 11:58:50.960081 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:50.959745    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt" event={"ID":"b3f9e19a-8336-4fe1-8ac3-721dcf430675","Type":"ContainerStarted","Data":"194ffd51bca9263e6544764e795c037034c1008acddde42f747431c26271ce16"}
Nov 03 11:58:50.960081 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:50.959772    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:51.071846 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:51.071797    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt" podStartSLOduration=2.071784416 podStartE2EDuration="2.071784416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:58:51.070652785 +0000 UTC m=+480.083119449" watchObservedRunningTime="2026-11-03 11:58:51.071784416 +0000 UTC m=+480.084251078"
Nov 03 11:58:51.965858 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:51.965829    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-klwdt"
Nov 03 11:58:52.168507 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.168477    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"]
Nov 03 11:58:52.168675 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.168640    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s" podUID="ab29b377-b610-482b-a8c0-817fdc612fc0" containerName="discovery" containerID="cri-o://2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9" gracePeriod=30
Nov 03 11:58:52.400023 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.400001    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:58:52.512449 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.512421    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-cacerts\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-cacerts\") pod \"ab29b377-b610-482b-a8c0-817fdc612fc0\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") "
Nov 03 11:58:52.512607 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.512464    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab29b377-b610-482b-a8c0-817fdc612fc0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab29b377-b610-482b-a8c0-817fdc612fc0-local-certs\") pod \"ab29b377-b610-482b-a8c0-817fdc612fc0\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") "
Nov 03 11:58:52.512607 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.512496    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-kubeconfig\") pod \"ab29b377-b610-482b-a8c0-817fdc612fc0\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") "
Nov 03 11:58:52.512607 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.512526    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-kube-api-access-wlzwh\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-kube-api-access-wlzwh\") pod \"ab29b377-b610-482b-a8c0-817fdc612fc0\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") "
Nov 03 11:58:52.512607 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.512552    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-token\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-token\") pod \"ab29b377-b610-482b-a8c0-817fdc612fc0\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") "
Nov 03 11:58:52.512607 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.512579    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-dns-cert\") pod \"ab29b377-b610-482b-a8c0-817fdc612fc0\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") "
Nov 03 11:58:52.513029 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.512618    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-ca-configmap\") pod \"ab29b377-b610-482b-a8c0-817fdc612fc0\" (UID: \"ab29b377-b610-482b-a8c0-817fdc612fc0\") "
Nov 03 11:58:52.513233 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.513192    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-ca-configmap" pod "ab29b377-b610-482b-a8c0-817fdc612fc0" (UID: "ab29b377-b610-482b-a8c0-817fdc612fc0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:58:52.515423 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.515383    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-kubeconfig" pod "ab29b377-b610-482b-a8c0-817fdc612fc0" (UID: "ab29b377-b610-482b-a8c0-817fdc612fc0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:58:52.515423 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.515405    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-kube-api-access-wlzwh" pod "ab29b377-b610-482b-a8c0-817fdc612fc0" (UID: "ab29b377-b610-482b-a8c0-817fdc612fc0"). InnerVolumeSpecName "kube-api-access-wlzwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:58:52.515732 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.515688    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-cacerts" pod "ab29b377-b610-482b-a8c0-817fdc612fc0" (UID: "ab29b377-b610-482b-a8c0-817fdc612fc0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:58:52.515799 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.515759    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab29b377-b610-482b-a8c0-817fdc612fc0-local-certs" pod "ab29b377-b610-482b-a8c0-817fdc612fc0" (UID: "ab29b377-b610-482b-a8c0-817fdc612fc0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:58:52.516735 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.516671    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-dns-cert" pod "ab29b377-b610-482b-a8c0-817fdc612fc0" (UID: "ab29b377-b610-482b-a8c0-817fdc612fc0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:58:52.516843 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.516795    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-token" pod "ab29b377-b610-482b-a8c0-817fdc612fc0" (UID: "ab29b377-b610-482b-a8c0-817fdc612fc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:58:52.613479 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.613437    4909 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-cacerts\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:52.613479 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.613475    4909 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab29b377-b610-482b-a8c0-817fdc612fc0-local-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:52.613479 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.613490    4909 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-kubeconfig\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:52.613690 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.613504    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlzwh\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-kube-api-access-wlzwh\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:52.613690 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.613517    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:52.613690 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.613528    4909 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-dns-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:52.613690 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.613539    4909 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab29b377-b610-482b-a8c0-817fdc612fc0-istio-csr-ca-configmap\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:52.970040 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.970008    4909 generic.go:356] "Generic (PLEG): container finished" podID="ab29b377-b610-482b-a8c0-817fdc612fc0" containerID="2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9" exitCode=0
Nov 03 11:58:52.970431 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.970074    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"
Nov 03 11:58:52.970431 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.970096    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s" event={"ID":"ab29b377-b610-482b-a8c0-817fdc612fc0","Type":"ContainerDied","Data":"2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9"}
Nov 03 11:58:52.970431 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.970179    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s" event={"ID":"ab29b377-b610-482b-a8c0-817fdc612fc0","Type":"ContainerDied","Data":"8f00ab9dc1db179cb9fbb20be2b9c14cbc4e872335efe9834427e36ee17fb0cf"}
Nov 03 11:58:52.970431 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.970206    4909 scope.go:122] "RemoveContainer" containerID="2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9"
Nov 03 11:58:52.979508 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.979491    4909 scope.go:122] "RemoveContainer" containerID="2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9"
Nov 03 11:58:52.979851 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:58:52.979829    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9\": container with ID starting with 2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9 not found: ID does not exist" containerID="2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9"
Nov 03 11:58:52.979936 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:52.979858    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9"} err="failed to get container status \"2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9\": rpc error: code = NotFound desc = could not find container \"2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9\": container with ID starting with 2033657dd72f449789cc53619cba84c9381966a97045885700091aca94c474e9 not found: ID does not exist"
Nov 03 11:58:53.122271 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:53.122245    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"]
Nov 03 11:58:53.161958 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:53.161937    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4hn7s"]
Nov 03 11:58:55.121105 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.121077    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab29b377-b610-482b-a8c0-817fdc612fc0" path="/var/lib/kubelet/pods/ab29b377-b610-482b-a8c0-817fdc612fc0/volumes"
Nov 03 11:58:55.511884 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.511850    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-csc6h"]
Nov 03 11:58:55.512365 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.512340    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ab29b377-b610-482b-a8c0-817fdc612fc0" containerName="discovery"
Nov 03 11:58:55.512365 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.512360    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab29b377-b610-482b-a8c0-817fdc612fc0" containerName="discovery"
Nov 03 11:58:55.512509 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.512419    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab29b377-b610-482b-a8c0-817fdc612fc0" containerName="discovery"
Nov 03 11:58:55.516485 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.516469    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.564892 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.564864    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 11:58:55.610708 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.610672    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 11:58:55.610880 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.610861    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 11:58:55.610935 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.610893    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pgfs9\""
Nov 03 11:58:55.612079 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.612059    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-csc6h"]
Nov 03 11:58:55.615382 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.615359    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-cbwwg"]
Nov 03 11:58:55.618706 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.618679    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.636124 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.636103    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khggs\" (UniqueName: \"kubernetes.io/projected/e135ffb3-aaf9-493b-b0ba-87d09909709e-kube-api-access-khggs\") pod \"model-serving-api-75bd89fdfc-csc6h\" (UID: \"e135ffb3-aaf9-493b-b0ba-87d09909709e\") " pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.636205 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.636148    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e135ffb3-aaf9-493b-b0ba-87d09909709e-tls-certs\") pod \"model-serving-api-75bd89fdfc-csc6h\" (UID: \"e135ffb3-aaf9-493b-b0ba-87d09909709e\") " pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.672900 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.672876    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cbwwg"]
Nov 03 11:58:55.692156 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.692137    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rcp7f\""
Nov 03 11:58:55.692252 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.692176    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 11:58:55.692252 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.692178    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 11:58:55.736596 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.736571    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprcz\" (UniqueName: \"kubernetes.io/projected/d66fb109-6507-43fe-8916-82bcc32b4ecc-kube-api-access-kprcz\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.736732 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.736614    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khggs\" (UniqueName: \"kubernetes.io/projected/e135ffb3-aaf9-493b-b0ba-87d09909709e-kube-api-access-khggs\") pod \"model-serving-api-75bd89fdfc-csc6h\" (UID: \"e135ffb3-aaf9-493b-b0ba-87d09909709e\") " pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.736732 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.736643    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d66fb109-6507-43fe-8916-82bcc32b4ecc-cert\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.736827 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.736741    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66fb109-6507-43fe-8916-82bcc32b4ecc-metrics-certs\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.736827 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.736773    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e135ffb3-aaf9-493b-b0ba-87d09909709e-tls-certs\") pod \"model-serving-api-75bd89fdfc-csc6h\" (UID: \"e135ffb3-aaf9-493b-b0ba-87d09909709e\") " pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.739146 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.739125    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e135ffb3-aaf9-493b-b0ba-87d09909709e-tls-certs\") pod \"model-serving-api-75bd89fdfc-csc6h\" (UID: \"e135ffb3-aaf9-493b-b0ba-87d09909709e\") " pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.789890 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.789820    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khggs\" (UniqueName: \"kubernetes.io/projected/e135ffb3-aaf9-493b-b0ba-87d09909709e-kube-api-access-khggs\") pod \"model-serving-api-75bd89fdfc-csc6h\" (UID: \"e135ffb3-aaf9-493b-b0ba-87d09909709e\") " pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.828580 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.828556    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:58:55.837848 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.837823    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprcz\" (UniqueName: \"kubernetes.io/projected/d66fb109-6507-43fe-8916-82bcc32b4ecc-kube-api-access-kprcz\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.837942 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.837879    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d66fb109-6507-43fe-8916-82bcc32b4ecc-cert\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.837942 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.837922    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66fb109-6507-43fe-8916-82bcc32b4ecc-metrics-certs\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.840294 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.840275    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66fb109-6507-43fe-8916-82bcc32b4ecc-metrics-certs\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.840458 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.840438    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d66fb109-6507-43fe-8916-82bcc32b4ecc-cert\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.895371 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.895323    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprcz\" (UniqueName: \"kubernetes.io/projected/d66fb109-6507-43fe-8916-82bcc32b4ecc-kube-api-access-kprcz\") pod \"odh-model-controller-769875db56-cbwwg\" (UID: \"d66fb109-6507-43fe-8916-82bcc32b4ecc\") " pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:55.931144 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:55.931118    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:58:56.191025 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:56.190937    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-csc6h"]
Nov 03 11:58:56.193107 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:58:56.193076    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode135ffb3_aaf9_493b_b0ba_87d09909709e.slice/crio-206509ef3cc974ec5038e79b9812ddfd8080534139f7f538695177556df2c7c5 WatchSource:0}: Error finding container 206509ef3cc974ec5038e79b9812ddfd8080534139f7f538695177556df2c7c5: Status 404 returned error can't find the container with id 206509ef3cc974ec5038e79b9812ddfd8080534139f7f538695177556df2c7c5
Nov 03 11:58:56.290083 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:56.290055    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cbwwg"]
Nov 03 11:58:56.292684 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:58:56.292655    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66fb109_6507_43fe_8916_82bcc32b4ecc.slice/crio-34e07ee2bbb67af2d822b23fa2c82d1f4c8f0057d7385e43e29f130979780767 WatchSource:0}: Error finding container 34e07ee2bbb67af2d822b23fa2c82d1f4c8f0057d7385e43e29f130979780767: Status 404 returned error can't find the container with id 34e07ee2bbb67af2d822b23fa2c82d1f4c8f0057d7385e43e29f130979780767
Nov 03 11:58:56.991300 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:56.991259    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cbwwg" event={"ID":"d66fb109-6507-43fe-8916-82bcc32b4ecc","Type":"ContainerStarted","Data":"34e07ee2bbb67af2d822b23fa2c82d1f4c8f0057d7385e43e29f130979780767"}
Nov 03 11:58:56.992336 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:58:56.992311    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-csc6h" event={"ID":"e135ffb3-aaf9-493b-b0ba-87d09909709e","Type":"ContainerStarted","Data":"206509ef3cc974ec5038e79b9812ddfd8080534139f7f538695177556df2c7c5"}
Nov 03 11:59:00.008317 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:00.008279    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-csc6h" event={"ID":"e135ffb3-aaf9-493b-b0ba-87d09909709e","Type":"ContainerStarted","Data":"ad9419e182c3078b79a632151874d8f6c859a05afd646b64b8039fe341d95e27"}
Nov 03 11:59:00.008797 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:00.008361    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:59:00.114186 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:00.114077    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-csc6h" podStartSLOduration=1.459364417 podStartE2EDuration="5.114058745s" totalImagesPullingTime="3.654694328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:00.113663826 +0000 UTC m=+489.126130491" watchObservedRunningTime="2026-11-03 11:59:00.114058745 +0000 UTC m=+489.126525410"
Nov 03 11:59:01.014039 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:01.013998    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cbwwg" event={"ID":"d66fb109-6507-43fe-8916-82bcc32b4ecc","Type":"ContainerStarted","Data":"7e385392de3b7a05b820e84536be6f23b28657e296959ef6be4b2d4842ec5473"}
Nov 03 11:59:01.014400 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:01.014072    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:59:01.122551 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:01.122501    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-cbwwg" podStartSLOduration=1.7663402289999999 podStartE2EDuration="6.122488527s" totalImagesPullingTime="4.356148298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:01.122191699 +0000 UTC m=+490.134658363" watchObservedRunningTime="2026-11-03 11:59:01.122488527 +0000 UTC m=+490.134955184"
Nov 03 11:59:08.705369 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.705334    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"]
Nov 03 11:59:08.708679 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.708654    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.736597 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.736570    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-metrics-certs\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.736715 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.736621    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aff0072-c198-4b62-a51f-48cfe2eda6df-openshift-service-ca\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.736715 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.736638    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-maas-api-tls\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.736824 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.736715    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2vr\" (UniqueName: \"kubernetes.io/projected/2aff0072-c198-4b62-a51f-48cfe2eda6df-kube-api-access-pl2vr\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.757655 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.757635    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 11:59:08.803646 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.803616    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 11:59:08.803646 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.803625    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 11:59:08.803820 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.803625    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vv49\""
Nov 03 11:59:08.803820 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.803731    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 11:59:08.805111 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.805092    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"]
Nov 03 11:59:08.837112 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.837090    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aff0072-c198-4b62-a51f-48cfe2eda6df-openshift-service-ca\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.837270 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.837116    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-maas-api-tls\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.837270 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.837131    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2vr\" (UniqueName: \"kubernetes.io/projected/2aff0072-c198-4b62-a51f-48cfe2eda6df-kube-api-access-pl2vr\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.837270 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.837173    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-metrics-certs\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.838265 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.838235    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aff0072-c198-4b62-a51f-48cfe2eda6df-openshift-service-ca\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.839737 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.839713    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-maas-api-tls\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.839936 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.839921    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-metrics-certs\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:08.889863 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:08.889837    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2vr\" (UniqueName: \"kubernetes.io/projected/2aff0072-c198-4b62-a51f-48cfe2eda6df-kube-api-access-pl2vr\") pod \"maas-api-5dff6fb7dd-p7xv4\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") " pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:09.020958 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:09.020891    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:09.382786 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:09.382561    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"]
Nov 03 11:59:09.385508 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:59:09.385477    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aff0072_c198_4b62_a51f_48cfe2eda6df.slice/crio-e8e161086389e4b4248b9d14fb48444cd989440b74530f522fc5b5fe31d478e5 WatchSource:0}: Error finding container e8e161086389e4b4248b9d14fb48444cd989440b74530f522fc5b5fe31d478e5: Status 404 returned error can't find the container with id e8e161086389e4b4248b9d14fb48444cd989440b74530f522fc5b5fe31d478e5
Nov 03 11:59:09.562648 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:59:09.562617    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 11:59:10.049801 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:10.049758    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4" event={"ID":"2aff0072-c198-4b62-a51f-48cfe2eda6df","Type":"ContainerStarted","Data":"e8e161086389e4b4248b9d14fb48444cd989440b74530f522fc5b5fe31d478e5"}
Nov 03 11:59:11.018908 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:11.018880    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-csc6h"
Nov 03 11:59:12.020857 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:12.020826    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-cbwwg"
Nov 03 11:59:13.898198 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:13.898165    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 11:59:20.091788 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:20.091753    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4" event={"ID":"2aff0072-c198-4b62-a51f-48cfe2eda6df","Type":"ContainerStarted","Data":"0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523"}
Nov 03 11:59:20.092149 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:20.091878    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:20.198212 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:20.198107    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4" podStartSLOduration=2.201708446 podStartE2EDuration="12.198090754s" totalImagesPullingTime="9.996382308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:20.196749348 +0000 UTC m=+509.209216011" watchObservedRunningTime="2026-11-03 11:59:20.198090754 +0000 UTC m=+509.210557418"
Nov 03 11:59:26.102298 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:26.102267    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:27.151243 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.151202    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"]
Nov 03 11:59:27.226216 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.226183    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"]
Nov 03 11:59:27.226363 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.226314    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.278356 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.278323    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbd0bd-681f-4db9-9ddc-1563d890c150-openshift-service-ca\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.278484 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.278376    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvlsj\" (UniqueName: \"kubernetes.io/projected/1adbd0bd-681f-4db9-9ddc-1563d890c150-kube-api-access-lvlsj\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.278484 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.278444    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-maas-api-tls\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.278556 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.278489    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-metrics-certs\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.379736 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.379690    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvlsj\" (UniqueName: \"kubernetes.io/projected/1adbd0bd-681f-4db9-9ddc-1563d890c150-kube-api-access-lvlsj\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.379892 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.379746    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-maas-api-tls\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.379892 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.379799    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-metrics-certs\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.379892 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.379838    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbd0bd-681f-4db9-9ddc-1563d890c150-openshift-service-ca\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.380491 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.380456    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbd0bd-681f-4db9-9ddc-1563d890c150-openshift-service-ca\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.382202 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.382183    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-metrics-certs\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.382296 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.382229    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-maas-api-tls\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.432800 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.432742    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvlsj\" (UniqueName: \"kubernetes.io/projected/1adbd0bd-681f-4db9-9ddc-1563d890c150-kube-api-access-lvlsj\") pod \"maas-api-6f554dd4c5-phtbh\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.558200 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.558167    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:27.936744 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:27.936718    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"]
Nov 03 11:59:27.939629 ip-10-0-131-136 kubenswrapper[4909]: W1103 11:59:27.939597    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbd0bd_681f_4db9_9ddc_1563d890c150.slice/crio-3ed5db4c0c3017f77856aa89a98cf8ff24ece7cfaa8ceecf7d5b93e1b9e38e1a WatchSource:0}: Error finding container 3ed5db4c0c3017f77856aa89a98cf8ff24ece7cfaa8ceecf7d5b93e1b9e38e1a: Status 404 returned error can't find the container with id 3ed5db4c0c3017f77856aa89a98cf8ff24ece7cfaa8ceecf7d5b93e1b9e38e1a
Nov 03 11:59:28.124769 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:28.124738    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh" event={"ID":"1adbd0bd-681f-4db9-9ddc-1563d890c150","Type":"ContainerStarted","Data":"3ed5db4c0c3017f77856aa89a98cf8ff24ece7cfaa8ceecf7d5b93e1b9e38e1a"}
Nov 03 11:59:29.129539 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:29.129458    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh" event={"ID":"1adbd0bd-681f-4db9-9ddc-1563d890c150","Type":"ContainerStarted","Data":"b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90"}
Nov 03 11:59:29.129939 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:29.129626    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:29.236460 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:29.236415    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh" podStartSLOduration=1.456265062 podStartE2EDuration="2.23640216s" totalImagesPullingTime="780.137098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:29.235752334 +0000 UTC m=+518.248218995" watchObservedRunningTime="2026-11-03 11:59:29.23640216 +0000 UTC m=+518.248868880"
Nov 03 11:59:35.139263 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.139232    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 11:59:35.292275 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.292247    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"]
Nov 03 11:59:35.292516 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.292475    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4" podUID="2aff0072-c198-4b62-a51f-48cfe2eda6df" containerName="maas-api" containerID="cri-o://0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523" gracePeriod=30
Nov 03 11:59:35.532783 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.532754    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:35.647912 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.647883    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-maas-api-tls\") pod \"2aff0072-c198-4b62-a51f-48cfe2eda6df\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") "
Nov 03 11:59:35.648068 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.647942    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-metrics-certs\") pod \"2aff0072-c198-4b62-a51f-48cfe2eda6df\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") "
Nov 03 11:59:35.648068 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.647971    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aff0072-c198-4b62-a51f-48cfe2eda6df-kube-api-access-pl2vr\" (UniqueName: \"kubernetes.io/projected/2aff0072-c198-4b62-a51f-48cfe2eda6df-kube-api-access-pl2vr\") pod \"2aff0072-c198-4b62-a51f-48cfe2eda6df\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") "
Nov 03 11:59:35.648068 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.647990    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aff0072-c198-4b62-a51f-48cfe2eda6df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aff0072-c198-4b62-a51f-48cfe2eda6df-openshift-service-ca\") pod \"2aff0072-c198-4b62-a51f-48cfe2eda6df\" (UID: \"2aff0072-c198-4b62-a51f-48cfe2eda6df\") "
Nov 03 11:59:35.648425 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.648401    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aff0072-c198-4b62-a51f-48cfe2eda6df-openshift-service-ca" pod "2aff0072-c198-4b62-a51f-48cfe2eda6df" (UID: "2aff0072-c198-4b62-a51f-48cfe2eda6df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:59:35.650202 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.650172    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aff0072-c198-4b62-a51f-48cfe2eda6df-kube-api-access-pl2vr" pod "2aff0072-c198-4b62-a51f-48cfe2eda6df" (UID: "2aff0072-c198-4b62-a51f-48cfe2eda6df"). InnerVolumeSpecName "kube-api-access-pl2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:59:35.650584 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.650561    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-metrics-certs" pod "2aff0072-c198-4b62-a51f-48cfe2eda6df" (UID: "2aff0072-c198-4b62-a51f-48cfe2eda6df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:59:35.650686 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.650588    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-maas-api-tls" pod "2aff0072-c198-4b62-a51f-48cfe2eda6df" (UID: "2aff0072-c198-4b62-a51f-48cfe2eda6df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:59:35.749664 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.749629    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aff0072-c198-4b62-a51f-48cfe2eda6df-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:59:35.749664 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.749666    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:59:35.749860 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.749681    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff0072-c198-4b62-a51f-48cfe2eda6df-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:59:35.749860 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:35.749722    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl2vr\" (UniqueName: \"kubernetes.io/projected/2aff0072-c198-4b62-a51f-48cfe2eda6df-kube-api-access-pl2vr\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:59:36.157607 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.157576    4909 generic.go:356] "Generic (PLEG): container finished" podID="2aff0072-c198-4b62-a51f-48cfe2eda6df" containerID="0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523" exitCode=0
Nov 03 11:59:36.158028 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.157623    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"
Nov 03 11:59:36.158028 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.157667    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4" event={"ID":"2aff0072-c198-4b62-a51f-48cfe2eda6df","Type":"ContainerDied","Data":"0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523"}
Nov 03 11:59:36.158028 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.157739    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4" event={"ID":"2aff0072-c198-4b62-a51f-48cfe2eda6df","Type":"ContainerDied","Data":"e8e161086389e4b4248b9d14fb48444cd989440b74530f522fc5b5fe31d478e5"}
Nov 03 11:59:36.158028 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.157769    4909 scope.go:122] "RemoveContainer" containerID="0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523"
Nov 03 11:59:36.166998 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.166974    4909 scope.go:122] "RemoveContainer" containerID="0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523"
Nov 03 11:59:36.167248 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:59:36.167232    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523\": container with ID starting with 0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523 not found: ID does not exist" containerID="0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523"
Nov 03 11:59:36.167299 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.167258    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523"} err="failed to get container status \"0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523\": rpc error: code = NotFound desc = could not find container \"0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523\": container with ID starting with 0ba29839876858cac4ba8fc8fe0dd675a4ba4b6f2af1f214ded5218d151f9523 not found: ID does not exist"
Nov 03 11:59:36.308227 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.308201    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"]
Nov 03 11:59:36.354926 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:36.354897    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dff6fb7dd-p7xv4"]
Nov 03 11:59:37.121370 ip-10-0-131-136 kubenswrapper[4909]: I1103 11:59:37.121335    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aff0072-c198-4b62-a51f-48cfe2eda6df" path="/var/lib/kubelet/pods/2aff0072-c198-4b62-a51f-48cfe2eda6df/volumes"
Nov 03 11:59:39.562021 ip-10-0-131-136 kubenswrapper[4909]: E1103 11:59:39.561982    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:00:09.564007 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:00:09.563904    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:00:39.562660 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:00:39.562587    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:00:51.157447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:00:51.157421    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:00:51.157938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:00:51.157496    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:01:09.563390 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:01:09.563347    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:01:31.903795 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:31.903766    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:01:33.831074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.831041    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"]
Nov 03 12:01:33.831462 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.831388    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2aff0072-c198-4b62-a51f-48cfe2eda6df" containerName="maas-api"
Nov 03 12:01:33.831462 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.831401    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aff0072-c198-4b62-a51f-48cfe2eda6df" containerName="maas-api"
Nov 03 12:01:33.831535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.831487    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aff0072-c198-4b62-a51f-48cfe2eda6df" containerName="maas-api"
Nov 03 12:01:33.834358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.834340    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:33.882890 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.882858    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 12:01:33.928651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.928616    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 12:01:33.928840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.928661    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8vc6k\""
Nov 03 12:01:33.928840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.928686    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 12:01:33.930036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.930014    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"]
Nov 03 12:01:33.937125 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.937104    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:33.937191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.937146    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rq76\" (UniqueName: \"kubernetes.io/projected/a893c515-62e2-4508-a564-127448f1d80f-kube-api-access-6rq76\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:33.937271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.937220    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:33.937312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.937287    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:33.937344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.937327    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:33.937390 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:33.937377    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a893c515-62e2-4508-a564-127448f1d80f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038261 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038226    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a893c515-62e2-4508-a564-127448f1d80f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038273    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038303    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rq76\" (UniqueName: \"kubernetes.io/projected/a893c515-62e2-4508-a564-127448f1d80f-kube-api-access-6rq76\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038330    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038352    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038372    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038839 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038804    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038815    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.038951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.038855    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.040514 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.040493    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a893c515-62e2-4508-a564-127448f1d80f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.040798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.040779    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a893c515-62e2-4508-a564-127448f1d80f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.091072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.091012    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rq76\" (UniqueName: \"kubernetes.io/projected/a893c515-62e2-4508-a564-127448f1d80f-kube-api-access-6rq76\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-l4w58\" (UID: \"a893c515-62e2-4508-a564-127448f1d80f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.146620 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.146594    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:34.509207 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.509170    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"]
Nov 03 12:01:34.512068 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:01:34.512032    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda893c515_62e2_4508_a564_127448f1d80f.slice/crio-ced32a198b9ec611674ab4fd0235aea91c19464bf797a3b5fc1da3edb2c26fd6 WatchSource:0}: Error finding container ced32a198b9ec611674ab4fd0235aea91c19464bf797a3b5fc1da3edb2c26fd6: Status 404 returned error can't find the container with id ced32a198b9ec611674ab4fd0235aea91c19464bf797a3b5fc1da3edb2c26fd6
Nov 03 12:01:34.623415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:34.623375    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58" event={"ID":"a893c515-62e2-4508-a564-127448f1d80f","Type":"ContainerStarted","Data":"ced32a198b9ec611674ab4fd0235aea91c19464bf797a3b5fc1da3edb2c26fd6"}
Nov 03 12:01:36.637554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:36.637517    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58" event={"ID":"a893c515-62e2-4508-a564-127448f1d80f","Type":"ContainerStarted","Data":"3c658b356e19e7b6867a4d65e1521869af7ad2b318e5c43e2f3342623a99084f"}
Nov 03 12:01:36.783796 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:36.783749    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58" podStartSLOduration=2.203007015 podStartE2EDuration="3.783734904s" totalImagesPullingTime="1.580727889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:36.783111119 +0000 UTC m=+645.795577783" watchObservedRunningTime="2026-11-03 12:01:36.783734904 +0000 UTC m=+645.796201566"
Nov 03 12:01:37.074915 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.074887    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"]
Nov 03 12:01:37.078468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.078444    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.127032 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.127005    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 12:01:37.147436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.147398    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:37.164043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.164017    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.164145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.164045    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.164145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.164060    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.164292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.164269    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7892de29-7641-4f9a-8e86-787c3e3c40bb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.164340 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.164316    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24mc4\" (UniqueName: \"kubernetes.io/projected/7892de29-7641-4f9a-8e86-787c3e3c40bb-kube-api-access-24mc4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.164453 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.164437    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.173967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.173944    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"]
Nov 03 12:01:37.264780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.264755    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.264878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.264785    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.264878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.264800    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.264878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.264829    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7892de29-7641-4f9a-8e86-787c3e3c40bb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.264878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.264854    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24mc4\" (UniqueName: \"kubernetes.io/projected/7892de29-7641-4f9a-8e86-787c3e3c40bb-kube-api-access-24mc4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.265027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.264914    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.265182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.265161    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.265260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.265235    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.265404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.265323    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.267125 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.267104    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7892de29-7641-4f9a-8e86-787c3e3c40bb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.267264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.267247    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7892de29-7641-4f9a-8e86-787c3e3c40bb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.317655 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.317630    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24mc4\" (UniqueName: \"kubernetes.io/projected/7892de29-7641-4f9a-8e86-787c3e3c40bb-kube-api-access-24mc4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9xcdv\" (UID: \"7892de29-7641-4f9a-8e86-787c3e3c40bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.390227 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.390170    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:37.753692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:37.753668    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"]
Nov 03 12:01:37.756005 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:01:37.755973    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7892de29_7641_4f9a_8e86_787c3e3c40bb.slice/crio-5cd6d7ea29761b9c007ea85de3577e4c55cb7e87db308bee803bf9b022894ddb WatchSource:0}: Error finding container 5cd6d7ea29761b9c007ea85de3577e4c55cb7e87db308bee803bf9b022894ddb: Status 404 returned error can't find the container with id 5cd6d7ea29761b9c007ea85de3577e4c55cb7e87db308bee803bf9b022894ddb
Nov 03 12:01:38.647164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:38.647089    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv" event={"ID":"7892de29-7641-4f9a-8e86-787c3e3c40bb","Type":"ContainerStarted","Data":"81de08ba7ba2317adfa07595acdba80069bf3f2baf6ae59d3a5ed26b08b8a81d"}
Nov 03 12:01:38.647164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:38.647145    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv" event={"ID":"7892de29-7641-4f9a-8e86-787c3e3c40bb","Type":"ContainerStarted","Data":"5cd6d7ea29761b9c007ea85de3577e4c55cb7e87db308bee803bf9b022894ddb"}
Nov 03 12:01:38.752250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:38.752202    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv" podStartSLOduration=1.149476508 podStartE2EDuration="1.752188371s" totalImagesPullingTime="602.711863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:38.752168788 +0000 UTC m=+647.764635431" watchObservedRunningTime="2026-11-03 12:01:38.752188371 +0000 UTC m=+647.764655030"
Nov 03 12:01:39.391320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:39.391291    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:39.562349 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:01:39.562312    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:01:42.897101 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:42.897069    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:01:44.040780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.040743    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"]
Nov 03 12:01:44.044372 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.044355    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.113148 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.113116    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.113294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.113156    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.113294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.113173    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.113294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.113228    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.113470 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.113291    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjk9\" (UniqueName: \"kubernetes.io/projected/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-kube-api-access-ffjk9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.113470 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.113372    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.138442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.138416    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 12:01:44.139796 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.139778    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"]
Nov 03 12:01:44.147571 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.147557    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:44.160147 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.160127    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:44.214032 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214012    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214053    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214069    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214089    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214159    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjk9\" (UniqueName: \"kubernetes.io/projected/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-kube-api-access-ffjk9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214325    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214484    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214578    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.214648 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.214603    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.216433 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.216413    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.216585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.216568    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.269853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.269828    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjk9\" (UniqueName: \"kubernetes.io/projected/6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582-kube-api-access-ffjk9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk\" (UID: \"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.356086 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.356019    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:44.689932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.689903    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-l4w58"
Nov 03 12:01:44.718017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:44.717989    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"]
Nov 03 12:01:44.721951 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:01:44.721926    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3dcfeb_8478_4ad9_b1cf_c2690a1a6582.slice/crio-3e361b55d8d5f06d7a0665d231f2513420d98b3cf1bcb26a6465548c08bca08b WatchSource:0}: Error finding container 3e361b55d8d5f06d7a0665d231f2513420d98b3cf1bcb26a6465548c08bca08b: Status 404 returned error can't find the container with id 3e361b55d8d5f06d7a0665d231f2513420d98b3cf1bcb26a6465548c08bca08b
Nov 03 12:01:45.683241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:45.683139    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk" event={"ID":"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582","Type":"ContainerStarted","Data":"f430bc690c30e449a109b91605c30bed9963450e479cdc6608d21ddd7c524e7b"}
Nov 03 12:01:45.683241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:45.683212    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk" event={"ID":"6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582","Type":"ContainerStarted","Data":"3e361b55d8d5f06d7a0665d231f2513420d98b3cf1bcb26a6465548c08bca08b"}
Nov 03 12:01:45.790520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:45.790412    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk" podStartSLOduration=1.167996431 podStartE2EDuration="1.790398597s" totalImagesPullingTime="622.402166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:45.789326805 +0000 UTC m=+654.801793448" watchObservedRunningTime="2026-11-03 12:01:45.790398597 +0000 UTC m=+654.802865260"
Nov 03 12:01:46.356551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:46.356518    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:47.390939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.390905    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:47.403239 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.403205    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:47.646263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.646192    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-f98p8"]
Nov 03 12:01:47.649569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.649552    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.706518 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.706486    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-f98p8"]
Nov 03 12:01:47.708014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.707994    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9xcdv"
Nov 03 12:01:47.743511 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.743482    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-home\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.743605 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.743516    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdpt\" (UniqueName: \"kubernetes.io/projected/1efe5333-ea36-4b1b-8369-e71b32c08e7c-kube-api-access-swdpt\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.743652 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.743635    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.743688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.743662    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.743688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.743680    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1efe5333-ea36-4b1b-8369-e71b32c08e7c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.743795 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.743780    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.757205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.757186    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 12:01:47.844981 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.844949    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1efe5333-ea36-4b1b-8369-e71b32c08e7c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.844990    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.845035    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-home\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.845055    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdpt\" (UniqueName: \"kubernetes.io/projected/1efe5333-ea36-4b1b-8369-e71b32c08e7c-kube-api-access-swdpt\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.845085    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.845106    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.845495    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845587 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.845529    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-home\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.845587 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.845554    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.847456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.847434    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1efe5333-ea36-4b1b-8369-e71b32c08e7c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.847787 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.847768    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1efe5333-ea36-4b1b-8369-e71b32c08e7c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.904294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.904225    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdpt\" (UniqueName: \"kubernetes.io/projected/1efe5333-ea36-4b1b-8369-e71b32c08e7c-kube-api-access-swdpt\") pod \"echo-simulator-kserve-64b6f8f96c-f98p8\" (UID: \"1efe5333-ea36-4b1b-8369-e71b32c08e7c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:47.962218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:47.962188    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:48.325463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:48.325445    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-f98p8"]
Nov 03 12:01:48.327628 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:01:48.327585    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efe5333_ea36_4b1b_8369_e71b32c08e7c.slice/crio-284c48f5f79cd23e632484cfef3d5fd6205f566ccc4f5369fa67457453114e4f WatchSource:0}: Error finding container 284c48f5f79cd23e632484cfef3d5fd6205f566ccc4f5369fa67457453114e4f: Status 404 returned error can't find the container with id 284c48f5f79cd23e632484cfef3d5fd6205f566ccc4f5369fa67457453114e4f
Nov 03 12:01:48.329090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:48.329074    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:01:48.698081 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:48.698047    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8" event={"ID":"1efe5333-ea36-4b1b-8369-e71b32c08e7c","Type":"ContainerStarted","Data":"284c48f5f79cd23e632484cfef3d5fd6205f566ccc4f5369fa67457453114e4f"}
Nov 03 12:01:51.714171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:51.714133    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8" event={"ID":"1efe5333-ea36-4b1b-8369-e71b32c08e7c","Type":"ContainerStarted","Data":"5c09fa12fef7286ad37bae17b65f34e0005b8cc8ecdac6cc080d5e115643d1ae"}
Nov 03 12:01:51.825959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:51.825907    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8" podStartSLOduration=2.31370486 podStartE2EDuration="4.825889876s" totalImagesPullingTime="2.512185016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:51.825841927 +0000 UTC m=+660.838308589" watchObservedRunningTime="2026-11-03 12:01:51.825889876 +0000 UTC m=+660.838356533"
Nov 03 12:01:51.963064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:51.963030    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:54.356886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:54.356843    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:54.369258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:54.369235    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:54.737016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:54.736990    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk"
Nov 03 12:01:57.631064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:57.631030    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:01:57.962607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:57.962570    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:57.975965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:57.975944    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:01:58.755353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:01:58.755326    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-f98p8"
Nov 03 12:02:02.709422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.709352    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"]
Nov 03 12:02:02.713796 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.713776    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.756238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.756214    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"]
Nov 03 12:02:02.760157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.760138    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.767297 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767278    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg57t\""
Nov 03 12:02:02.767548 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767523    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767612 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767565    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hsd\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-kube-api-access-k8hsd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767649    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767716    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e47f057-cdb7-45a5-99b9-4e16c5887078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767784    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767835 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767812    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767842    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767864    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.767972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.767947    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.809658 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.809631    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"]
Nov 03 12:02:02.818512 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.818490    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"]
Nov 03 12:02:02.868534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868506    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.868687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868551    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.868687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868586    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.868687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868625    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.868687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868676    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.868931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868771    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.868931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868807    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.868931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868828    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.868931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868898    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.869133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868968    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.869133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.868999    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869026    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hsd\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-kube-api-access-k8hsd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869042    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869057    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869097    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.869449 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869218    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869449 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869298    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869462    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869513    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.869672 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869564    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e47f057-cdb7-45a5-99b9-4e16c5887078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.869672 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869633    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.869799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.869671    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlv6\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-kube-api-access-stlv6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.870209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.870187    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e47f057-cdb7-45a5-99b9-4e16c5887078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.871138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.871120    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.871252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.871234    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.921923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.921900    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.922090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.922073    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hsd\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-kube-api-access-k8hsd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sdw94\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:02.970455 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970402    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970455 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970450    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970481    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970500    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970518    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970541    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970568    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlv6\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-kube-api-access-stlv6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970605    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970648    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.970974 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970905    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.971031 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.970983    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.971087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.971035    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.971127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.971098    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.971234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.971210    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.972778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.972755    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:02.972980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:02.972959    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:03.023322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.023294    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:03.023399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.023343    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlv6\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-kube-api-access-stlv6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cgbnn\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:03.026184 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.026166    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:03.072560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.072206    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:03.434422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.434401    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"]
Nov 03 12:02:03.436263 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:02:03.436232    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e47f057_cdb7_45a5_99b9_4e16c5887078.slice/crio-e474b9588b845dc07f7cda1a44b1eec7d5e4142809da8b66124d8fcbd395e56b WatchSource:0}: Error finding container e474b9588b845dc07f7cda1a44b1eec7d5e4142809da8b66124d8fcbd395e56b: Status 404 returned error can't find the container with id e474b9588b845dc07f7cda1a44b1eec7d5e4142809da8b66124d8fcbd395e56b
Nov 03 12:02:03.486738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.486716    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"]
Nov 03 12:02:03.489263 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:02:03.489240    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7f021b_e65f_4a54_ad09_69471a8d9b0a.slice/crio-53d8b3765df42951ca41f1123d8400421b037cd2b7e6d03aeb4f40019f456fb5 WatchSource:0}: Error finding container 53d8b3765df42951ca41f1123d8400421b037cd2b7e6d03aeb4f40019f456fb5: Status 404 returned error can't find the container with id 53d8b3765df42951ca41f1123d8400421b037cd2b7e6d03aeb4f40019f456fb5
Nov 03 12:02:03.765138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.765106    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" event={"ID":"0a7f021b-e65f-4a54-ad09-69471a8d9b0a","Type":"ContainerStarted","Data":"53d8b3765df42951ca41f1123d8400421b037cd2b7e6d03aeb4f40019f456fb5"}
Nov 03 12:02:03.766324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:03.766295    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" event={"ID":"3e47f057-cdb7-45a5-99b9-4e16c5887078","Type":"ContainerStarted","Data":"e474b9588b845dc07f7cda1a44b1eec7d5e4142809da8b66124d8fcbd395e56b"}
Nov 03 12:02:07.588597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:07.588565    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:02:07.588999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:07.588631    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:02:07.588999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:07.588659    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:02:07.786646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:07.786601    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" event={"ID":"3e47f057-cdb7-45a5-99b9-4e16c5887078","Type":"ContainerStarted","Data":"bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d"}
Nov 03 12:02:07.945237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:07.945176    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podStartSLOduration=1.795691881 podStartE2EDuration="5.945161085s" totalImagesPullingTime="4.149469204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:02:07.942564821 +0000 UTC m=+676.955031484" watchObservedRunningTime="2026-11-03 12:02:07.945161085 +0000 UTC m=+676.957627747"
Nov 03 12:02:08.026992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.026958    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:08.028891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.028859    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:08.029044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.028954    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:08.454738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.454495    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:02:08.454738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.454591    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:02:08.454738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.454642    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:02:08.793970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.793927    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" event={"ID":"0a7f021b-e65f-4a54-ad09-69471a8d9b0a","Type":"ContainerStarted","Data":"17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130"}
Nov 03 12:02:08.908099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:08.908040    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podStartSLOduration=1.945495542 podStartE2EDuration="6.90802379s" totalImagesPullingTime="4.962528248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:02:08.906044369 +0000 UTC m=+677.918511031" watchObservedRunningTime="2026-11-03 12:02:08.90802379 +0000 UTC m=+677.920490453"
Nov 03 12:02:09.027189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:09.027148    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:09.027388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:09.027207    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:09.073121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:09.073036    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:09.074711 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:09.074666    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:09.074848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:09.074740    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:09.608810 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:02:09.608775    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:02:10.026921 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:10.026883    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:10.027339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:10.026936    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:10.076726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:10.075104    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:10.076726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:10.075165    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:11.027048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:11.027013    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:11.027452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:11.027067    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:11.075723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:11.075038    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:11.075723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:11.075092    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:12.027222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:12.027182    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:12.027601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:12.027248    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:12.074124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:12.074088    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:12.074391 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:12.074361    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:13.027504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:13.027465    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:13.027952 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:13.027526    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:13.073120 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:13.073079    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:13.073399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:13.073369    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:14.026800 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:14.026760    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:14.026972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:14.026840    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:14.072993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:14.072955    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:14.073359 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:14.073006    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:15.026998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:15.026957    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:02:15.027202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:15.027014    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 12:02:15.073717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:15.073670    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:15.074165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:15.073738    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:16.073412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:16.073374    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:02:16.073576 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:16.073439    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 12:02:16.126414 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:16.126383    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:16.127193 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:16.127168    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:16.127868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:16.127849    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:02:17.101284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:17.101245    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:17.101625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:17.101603    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:17.102447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:02:17.102427    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:02:39.563642 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:02:39.563605    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:03:06.944949 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:06.944913    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"]
Nov 03 12:03:06.948591 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:06.948568    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:06.996912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:06.996872    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1319e7ef-openshift-default-dockercfg-m5h96\""
Nov 03 12:03:07.044536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.044502    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"]
Nov 03 12:03:07.054814 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.054785    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-credential-socket\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.054950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.054839    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-socket\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.054950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.054859    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-certs\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.055029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.054975    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de78094-d888-4fc0-8a19-f781d7a898ac-istiod-ca-cert\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.055029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.055019    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-envoy\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.055147 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.055126    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-data\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.055207 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.055173    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-token\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.055271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.055202    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4xz\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-kube-api-access-fn4xz\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.055319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.055282    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-podinfo\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156466 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156432    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-credential-socket\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156485    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-socket\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156504    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-certs\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156529    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de78094-d888-4fc0-8a19-f781d7a898ac-istiod-ca-cert\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156559    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-envoy\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156581    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-data\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156602    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-token\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.156633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156621    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4xz\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-kube-api-access-fn4xz\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.157000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156661    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-podinfo\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.157000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.156911    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-socket\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.157103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.157015    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-certs\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.157103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.157074    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-credential-socket\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.157221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.157206    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-data\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.157343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.157325    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de78094-d888-4fc0-8a19-f781d7a898ac-istiod-ca-cert\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.159238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.159208    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-envoy\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.159321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.159273    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-podinfo\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.211693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.211591    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4xz\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-kube-api-access-fn4xz\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.224283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.224218    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-token\") pod \"e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") " pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.261975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.261943    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:07.499589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.499514    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"]
Nov 03 12:03:07.503389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.503369    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.551832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.551804    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-30e6ec98-gw-openshift-default-dockercfg-tk5cp\""
Nov 03 12:03:07.560436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.560406    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"]
Nov 03 12:03:07.661780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.661748    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-credential-socket\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.661948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.661792    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cn77\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-kube-api-access-6cn77\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.661948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.661875    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-envoy\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.661948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.661909    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-podinfo\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.662083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.661991    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-token\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.662083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.662013    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-data\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.662083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.662053    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-socket\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.662245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.662140    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-certs\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.662304 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.662241    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istiod-ca-cert\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.713018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.712861    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"]
Nov 03 12:03:07.715961 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:03:07.715932    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de78094_d888_4fc0_8a19_f781d7a898ac.slice/crio-0ead8414d2bbd610d1512d8149215b309d81afd3a24efcceee19e573dd512439 WatchSource:0}: Error finding container 0ead8414d2bbd610d1512d8149215b309d81afd3a24efcceee19e573dd512439: Status 404 returned error can't find the container with id 0ead8414d2bbd610d1512d8149215b309d81afd3a24efcceee19e573dd512439
Nov 03 12:03:07.718181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.718145    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:07.718288 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.718228    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:07.718348 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.718268    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:07.762741 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762718    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cn77\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-kube-api-access-6cn77\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.762832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762750    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-envoy\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.762832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762782    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-podinfo\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.762832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762816    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-token\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.762993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762842    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-data\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.762993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762894    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-socket\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.762993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762926    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-certs\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.762993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.762970    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istiod-ca-cert\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.763205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.763003    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-credential-socket\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.763355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.763328    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-socket\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.763466 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.763354    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-data\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.763530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.763464    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-credential-socket\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.763587 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.763538    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-certs\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.763692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.763673    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istiod-ca-cert\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.765095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.765074    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-envoy\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.765321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.765303    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-podinfo\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.824014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.823977    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cn77\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-kube-api-access-6cn77\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:07.824325 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:07.824309    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-token\") pod \"e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") " pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:08.053419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.053336    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" event={"ID":"4de78094-d888-4fc0-8a19-f781d7a898ac","Type":"ContainerStarted","Data":"ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9"}
Nov 03 12:03:08.053419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.053390    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" event={"ID":"4de78094-d888-4fc0-8a19-f781d7a898ac","Type":"ContainerStarted","Data":"0ead8414d2bbd610d1512d8149215b309d81afd3a24efcceee19e573dd512439"}
Nov 03 12:03:08.116008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.115978    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:08.164476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.163808    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" podStartSLOduration=2.163787907 podStartE2EDuration="2.163787907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:08.161221829 +0000 UTC m=+737.173688493" watchObservedRunningTime="2026-11-03 12:03:08.163787907 +0000 UTC m=+737.176254573"
Nov 03 12:03:08.262621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.262586    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:08.267300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.267277    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:08.481834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.481802    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"]
Nov 03 12:03:08.484067 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:03:08.484035    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf806bab1_f23a_4ca3_865f_ea5b34aa2d49.slice/crio-7056b6153acc496c3b83bbb11defa7581b9c14e2cb796c6f25b32a96061c1d14 WatchSource:0}: Error finding container 7056b6153acc496c3b83bbb11defa7581b9c14e2cb796c6f25b32a96061c1d14: Status 404 returned error can't find the container with id 7056b6153acc496c3b83bbb11defa7581b9c14e2cb796c6f25b32a96061c1d14
Nov 03 12:03:08.486457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.486428    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:08.486541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.486489    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:08.486541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:08.486515    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:09.059653 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:09.059615    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" event={"ID":"f806bab1-f23a-4ca3-865f-ea5b34aa2d49","Type":"ContainerStarted","Data":"208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960"}
Nov 03 12:03:09.060162 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:09.059672    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" event={"ID":"f806bab1-f23a-4ca3-865f-ea5b34aa2d49","Type":"ContainerStarted","Data":"7056b6153acc496c3b83bbb11defa7581b9c14e2cb796c6f25b32a96061c1d14"}
Nov 03 12:03:09.060162 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:09.059990    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:09.060911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:09.060894    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:09.127513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:09.127480    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:09.128888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:09.128862    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:09.169953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:09.169908    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podStartSLOduration=2.169894153 podStartE2EDuration="2.169894153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:09.167810812 +0000 UTC m=+738.180277470" watchObservedRunningTime="2026-11-03 12:03:09.169894153 +0000 UTC m=+738.182360815"
Nov 03 12:03:09.563993 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:09.563958    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:03:10.063879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:10.063831    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:10.064669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:10.064649    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:14.511849 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.511819    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"]
Nov 03 12:03:14.520246 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.520217    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:14.569794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.569752    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"]
Nov 03 12:03:14.613674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.613643    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-30e6ec98\""
Nov 03 12:03:14.613674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.613643    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-30e6ec98-dockercfg-djlbx\""
Nov 03 12:03:14.647594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.647564    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:14.647736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.647623    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hgk\" (UniqueName: \"kubernetes.io/projected/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-kube-api-access-g5hgk\") pod \"payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:14.748333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.748302    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:14.748333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.748340    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hgk\" (UniqueName: \"kubernetes.io/projected/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-kube-api-access-g5hgk\") pod \"payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:14.749134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.749112    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:14.801569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.801503    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hgk\" (UniqueName: \"kubernetes.io/projected/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-kube-api-access-g5hgk\") pod \"payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:14.833088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:14.833062    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:15.194158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.194134    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"]
Nov 03 12:03:15.196858 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:03:15.196831    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60616d0a_9b4c_473d_9ec9_1ae2efff8fa4.slice/crio-be6bbbee2cc934727dec7664e1a931e881f7dcbe1d5361bbc8e1ec641b5dd4b8 WatchSource:0}: Error finding container be6bbbee2cc934727dec7664e1a931e881f7dcbe1d5361bbc8e1ec641b5dd4b8: Status 404 returned error can't find the container with id be6bbbee2cc934727dec7664e1a931e881f7dcbe1d5361bbc8e1ec641b5dd4b8
Nov 03 12:03:15.913995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.913964    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"]
Nov 03 12:03:15.918872 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.918850    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:15.957551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.957529    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-metrics-certs\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:15.957651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.957559    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74160872-48f5-47de-94fe-01b5ba2afa8e-openshift-service-ca\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:15.957651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.957583    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:15.957812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.957785    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7d4h\" (UniqueName: \"kubernetes.io/projected/74160872-48f5-47de-94fe-01b5ba2afa8e-kube-api-access-x7d4h\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:15.975782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:15.975765    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f732077f-serving-cert\""
Nov 03 12:03:16.019420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.019396    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"]
Nov 03 12:03:16.058392 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.058360    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7d4h\" (UniqueName: \"kubernetes.io/projected/74160872-48f5-47de-94fe-01b5ba2afa8e-kube-api-access-x7d4h\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.058500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.058475    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-metrics-certs\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.058552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.058509    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74160872-48f5-47de-94fe-01b5ba2afa8e-openshift-service-ca\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.058552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.058537    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.058994 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:16.058966    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-serving-cert: secret "maas-api-e2e-ait-f732077f-serving-cert" not found
Nov 03 12:03:16.059097 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:16.059032    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls podName:74160872-48f5-47de-94fe-01b5ba2afa8e nodeName:}" failed. No retries permitted until 2026-11-03 12:03:16.55901646 +0000 UTC m=+745.571483104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls") pod "maas-api-e2e-ait-f732077f-5b95797745-qs6hs" (UID: "74160872-48f5-47de-94fe-01b5ba2afa8e") : secret "maas-api-e2e-ait-f732077f-serving-cert" not found
Nov 03 12:03:16.059597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.059574    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74160872-48f5-47de-94fe-01b5ba2afa8e-openshift-service-ca\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.061777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.061753    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-metrics-certs\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.090536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.090507    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j" event={"ID":"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4","Type":"ContainerStarted","Data":"be6bbbee2cc934727dec7664e1a931e881f7dcbe1d5361bbc8e1ec641b5dd4b8"}
Nov 03 12:03:16.113830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.113804    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7d4h\" (UniqueName: \"kubernetes.io/projected/74160872-48f5-47de-94fe-01b5ba2afa8e-kube-api-access-x7d4h\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.562276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.562231    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.565962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.565931    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls\") pod \"maas-api-e2e-ait-f732077f-5b95797745-qs6hs\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:16.832031 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:16.831944    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:17.204906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:17.204879    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"]
Nov 03 12:03:17.207043 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:03:17.207018    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74160872_48f5_47de_94fe_01b5ba2afa8e.slice/crio-e3a987bffb89d91d84cc8122110e0c38ddfd440967328fb155d7af0a5f709888 WatchSource:0}: Error finding container e3a987bffb89d91d84cc8122110e0c38ddfd440967328fb155d7af0a5f709888: Status 404 returned error can't find the container with id e3a987bffb89d91d84cc8122110e0c38ddfd440967328fb155d7af0a5f709888
Nov 03 12:03:18.102783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:18.102754    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs" event={"ID":"74160872-48f5-47de-94fe-01b5ba2afa8e","Type":"ContainerStarted","Data":"e3a987bffb89d91d84cc8122110e0c38ddfd440967328fb155d7af0a5f709888"}
Nov 03 12:03:19.108344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:19.108296    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j" event={"ID":"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4","Type":"ContainerStarted","Data":"a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7"}
Nov 03 12:03:19.108838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:19.108439    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:19.109666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:19.109643    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs" event={"ID":"74160872-48f5-47de-94fe-01b5ba2afa8e","Type":"ContainerStarted","Data":"0b771e2efec673c66c259e0da3b4c3e88aa8f37d64e6bf9c253f9ec8e1c52198"}
Nov 03 12:03:19.109794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:19.109688    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:19.213828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:19.213776    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j" podStartSLOduration=2.329848354 podStartE2EDuration="5.213763885s" totalImagesPullingTime="2.883915531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:19.213178577 +0000 UTC m=+748.225645464" watchObservedRunningTime="2026-11-03 12:03:19.213763885 +0000 UTC m=+748.226230547"
Nov 03 12:03:19.322259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:19.322193    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs" podStartSLOduration=3.126021195 podStartE2EDuration="4.322171754s" totalImagesPullingTime="1.196150559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:19.320238631 +0000 UTC m=+748.332705295" watchObservedRunningTime="2026-11-03 12:03:19.322171754 +0000 UTC m=+748.334638421"
Nov 03 12:03:20.669176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.669141    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"]
Nov 03 12:03:20.672894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.672877    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.700442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.700414    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094b2f92-4824-43ca-92dd-400c1d57ce3b-openshift-service-ca\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.700442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.700443    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-metrics-certs\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.700610 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.700465    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.700610 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.700586    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc72j\" (UniqueName: \"kubernetes.io/projected/094b2f92-4824-43ca-92dd-400c1d57ce3b-kube-api-access-pc72j\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.721116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.721092    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4b9370d1-serving-cert\""
Nov 03 12:03:20.777206 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.777172    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"]
Nov 03 12:03:20.801974 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.801937    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc72j\" (UniqueName: \"kubernetes.io/projected/094b2f92-4824-43ca-92dd-400c1d57ce3b-kube-api-access-pc72j\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.802124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.802081    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094b2f92-4824-43ca-92dd-400c1d57ce3b-openshift-service-ca\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.802124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.802109    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-metrics-certs\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.802197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.802143    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.802270 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:20.802257    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-serving-cert: secret "maas-api-e2e-ait-4b9370d1-serving-cert" not found
Nov 03 12:03:20.802332 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:20.802323    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls podName:094b2f92-4824-43ca-92dd-400c1d57ce3b nodeName:}" failed. No retries permitted until 2026-11-03 12:03:21.302302539 +0000 UTC m=+750.314769194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls") pod "maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8" (UID: "094b2f92-4824-43ca-92dd-400c1d57ce3b") : secret "maas-api-e2e-ait-4b9370d1-serving-cert" not found
Nov 03 12:03:20.802779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.802756    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094b2f92-4824-43ca-92dd-400c1d57ce3b-openshift-service-ca\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.804638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.804612    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-metrics-certs\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:20.858456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:20.858425    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc72j\" (UniqueName: \"kubernetes.io/projected/094b2f92-4824-43ca-92dd-400c1d57ce3b-kube-api-access-pc72j\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:21.307763 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:21.307725    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:21.310121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:21.310092    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls\") pod \"maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:21.585790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:21.585705    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:21.953793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:21.953768    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"]
Nov 03 12:03:22.125377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:22.125338    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8" event={"ID":"094b2f92-4824-43ca-92dd-400c1d57ce3b","Type":"ContainerStarted","Data":"8697c7b9238b01433f61c5385c737577bcca0a01b40f4a44de7555648da314ad"}
Nov 03 12:03:24.135797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:24.135766    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8" event={"ID":"094b2f92-4824-43ca-92dd-400c1d57ce3b","Type":"ContainerStarted","Data":"df421900cd5263ba4f7274c35cbceb3f02159c018a984f20dba385e1c8c8e372"}
Nov 03 12:03:24.136157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:24.135882    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:24.268285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:24.268232    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8" podStartSLOduration=2.877789155 podStartE2EDuration="4.268217901s" totalImagesPullingTime="1.390428746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:24.266963874 +0000 UTC m=+753.279430548" watchObservedRunningTime="2026-11-03 12:03:24.268217901 +0000 UTC m=+753.280684563"
Nov 03 12:03:25.134813 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.134767    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:03:25.309668 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.309638    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:03:25.453253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.453222    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"]
Nov 03 12:03:25.457118 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.457095    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.506358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.506333    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-30e6ec98-serving-cert\""
Nov 03 12:03:25.512095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.512070    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"]
Nov 03 12:03:25.543393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.543369    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rllj\" (UniqueName: \"kubernetes.io/projected/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-kube-api-access-7rllj\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.543482 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.543433    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-metrics-certs\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.543518 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.543484    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.543518 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.543513    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-openshift-service-ca\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.644354 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.644325    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-metrics-certs\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.644487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.644362    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.644487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.644378    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-openshift-service-ca\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.644487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.644433    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rllj\" (UniqueName: \"kubernetes.io/projected/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-kube-api-access-7rllj\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.644598 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:25.644491    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-serving-cert: secret "maas-api-e2e-migrate-30e6ec98-serving-cert" not found
Nov 03 12:03:25.644598 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:25.644576    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls podName:947f99e7-60d5-45bf-8ffa-b2a34b30a5c8 nodeName:}" failed. No retries permitted until 2026-11-03 12:03:26.144556279 +0000 UTC m=+755.157022920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls") pod "maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7" (UID: "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8") : secret "maas-api-e2e-migrate-30e6ec98-serving-cert" not found
Nov 03 12:03:25.645149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.645127    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-openshift-service-ca\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.646721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.646676    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-metrics-certs\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:25.701037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:25.701006    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rllj\" (UniqueName: \"kubernetes.io/projected/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-kube-api-access-7rllj\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:26.153447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:26.153415    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:26.155799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:26.155779    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls\") pod \"maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:26.369769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:26.369738    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:26.733476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:26.733451    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"]
Nov 03 12:03:26.735937 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:03:26.735910    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947f99e7_60d5_45bf_8ffa_b2a34b30a5c8.slice/crio-27414e6474a6a9cfe9cbad40213fc2d0b133b256e6d24cb9b7c097a5bb514dd9 WatchSource:0}: Error finding container 27414e6474a6a9cfe9cbad40213fc2d0b133b256e6d24cb9b7c097a5bb514dd9: Status 404 returned error can't find the container with id 27414e6474a6a9cfe9cbad40213fc2d0b133b256e6d24cb9b7c097a5bb514dd9
Nov 03 12:03:27.153076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:27.152945    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7" event={"ID":"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8","Type":"ContainerStarted","Data":"27414e6474a6a9cfe9cbad40213fc2d0b133b256e6d24cb9b7c097a5bb514dd9"}
Nov 03 12:03:28.159258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:28.159221    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7" event={"ID":"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8","Type":"ContainerStarted","Data":"3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc"}
Nov 03 12:03:28.159639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:28.159311    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:28.266056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:28.265991    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7" podStartSLOduration=2.321109061 podStartE2EDuration="3.265976235s" totalImagesPullingTime="944.867174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:28.265112865 +0000 UTC m=+757.277579529" watchObservedRunningTime="2026-11-03 12:03:28.265976235 +0000 UTC m=+757.278442898"
Nov 03 12:03:30.118245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:30.118213    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:03:30.147622 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:30.147593    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:03:32.101151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:32.101069    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:03:34.170316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.170287    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:03:34.270906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.270873    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"]
Nov 03 12:03:34.274316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.274295    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.319117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.319090    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.319262 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.319124    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-metrics-certs\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.319262 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.319251    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cv2\" (UniqueName: \"kubernetes.io/projected/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-kube-api-access-h4cv2\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.319341 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.319280    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-openshift-service-ca\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.322568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.322552    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1319e7ef-serving-cert\""
Nov 03 12:03:34.329642 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.329591    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"]
Nov 03 12:03:34.420249 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.420217    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.420450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.420255    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-metrics-certs\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.420450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.420349    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cv2\" (UniqueName: \"kubernetes.io/projected/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-kube-api-access-h4cv2\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.420450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.420383    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-openshift-service-ca\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.420450 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:34.420351    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-serving-cert: secret "maas-api-e2e-ait-1319e7ef-serving-cert" not found
Nov 03 12:03:34.420725 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:34.420485    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls podName:29c3188f-a6a0-4215-beb4-d89e60c9b0fa nodeName:}" failed. No retries permitted until 2026-11-03 12:03:34.920464708 +0000 UTC m=+763.932931348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls") pod "maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" (UID: "29c3188f-a6a0-4215-beb4-d89e60c9b0fa") : secret "maas-api-e2e-ait-1319e7ef-serving-cert" not found
Nov 03 12:03:34.421066 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.421045    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-openshift-service-ca\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.422571 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.422547    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-metrics-certs\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.476282 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.476252    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cv2\" (UniqueName: \"kubernetes.io/projected/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-kube-api-access-h4cv2\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.924523 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.924486    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:34.926968 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:34.926951    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls\") pod \"maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:35.186872 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:35.186787    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:35.567518 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:35.567476    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"]
Nov 03 12:03:35.574170 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:03:35.574144    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c3188f_a6a0_4215_beb4_d89e60c9b0fa.slice/crio-4ab7a2694761538d27556e634205b0b6d31840b4471b073d8158484f077b2b42 WatchSource:0}: Error finding container 4ab7a2694761538d27556e634205b0b6d31840b4471b073d8158484f077b2b42: Status 404 returned error can't find the container with id 4ab7a2694761538d27556e634205b0b6d31840b4471b073d8158484f077b2b42
Nov 03 12:03:36.195684 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:36.195633    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" event={"ID":"29c3188f-a6a0-4215-beb4-d89e60c9b0fa","Type":"ContainerStarted","Data":"4ab7a2694761538d27556e634205b0b6d31840b4471b073d8158484f077b2b42"}
Nov 03 12:03:37.202095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:37.202058    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" event={"ID":"29c3188f-a6a0-4215-beb4-d89e60c9b0fa","Type":"ContainerStarted","Data":"32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03"}
Nov 03 12:03:37.202524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:37.202135    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:37.314844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:37.314792    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" podStartSLOduration=2.454481453 podStartE2EDuration="3.314777513s" totalImagesPullingTime="860.29606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:37.313664045 +0000 UTC m=+766.326130708" watchObservedRunningTime="2026-11-03 12:03:37.314777513 +0000 UTC m=+766.327244176"
Nov 03 12:03:39.564040 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:03:39.564001    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:03:41.105850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:41.105815    4909 patch_prober.go:29] interesting pod/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:03:41.106328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:41.105871    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:03:42.236503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:42.236438    4909 generic.go:356] "Generic (PLEG): container finished" podID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerID="208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960" exitCode=0
Nov 03 12:03:42.236503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:42.236482    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" event={"ID":"f806bab1-f23a-4ca3-865f-ea5b34aa2d49","Type":"ContainerDied","Data":"208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960"}
Nov 03 12:03:42.236923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:42.236834    4909 scope.go:122] "RemoveContainer" containerID="208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960"
Nov 03 12:03:42.238035 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:42.238002    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:42.238143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:42.238076    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:42.238143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:42.238108    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:43.213353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:43.213326    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:03:43.290625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:43.290592    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" event={"ID":"f806bab1-f23a-4ca3-865f-ea5b34aa2d49","Type":"ContainerStarted","Data":"423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42"}
Nov 03 12:03:44.116345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:44.116301    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:44.116552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:44.116517    4909 patch_prober.go:29] interesting pod/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 12:03:44.116632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:44.116570    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 12:03:45.116805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:45.116756    4909 patch_prober.go:29] interesting pod/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 12:03:45.117223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:45.116820    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 12:03:46.116424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:46.116388    4909 patch_prober.go:29] interesting pod/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 12:03:46.116600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:46.116438    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 12:03:47.116640 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:47.116603    4909 patch_prober.go:29] interesting pod/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 12:03:47.117140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:47.116653    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 12:03:48.117262 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:48.117227    4909 patch_prober.go:29] interesting pod/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 12:03:48.117626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:48.117288    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 12:03:49.124512 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:49.124485    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:49.124933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:49.124857    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:49.125527 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:49.125504    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:03:49.541438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:49.541404    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:03:51.335330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:51.335232    4909 generic.go:356] "Generic (PLEG): container finished" podID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerID="ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9" exitCode=0
Nov 03 12:03:51.335330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:51.335310    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" event={"ID":"4de78094-d888-4fc0-8a19-f781d7a898ac","Type":"ContainerDied","Data":"ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9"}
Nov 03 12:03:51.335817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:51.335633    4909 scope.go:122] "RemoveContainer" containerID="ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9"
Nov 03 12:03:51.336755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:51.336733    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:51.336805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:51.336795    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:51.336841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:51.336828    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:03:52.356327 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:52.356284    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" event={"ID":"4de78094-d888-4fc0-8a19-f781d7a898ac","Type":"ContainerStarted","Data":"8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed"}
Nov 03 12:03:53.262886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:53.262851    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:53.263821 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:53.263793    4909 patch_prober.go:29] interesting pod/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 12:03:53.263929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:53.263840    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 12:03:54.060724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:54.059793    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:54.263044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:54.263010    4909 patch_prober.go:29] interesting pod/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 12:03:54.263210 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:54.263067    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 12:03:55.263100 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:55.263066    4909 patch_prober.go:29] interesting pod/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 12:03:55.263498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:55.263130    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 12:03:56.263063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:56.263022    4909 patch_prober.go:29] interesting pod/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 12:03:56.263265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:56.263087    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 12:03:57.262969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:57.262932    4909 patch_prober.go:29] interesting pod/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 12:03:57.263145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:57.262987    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 12:03:58.266601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:58.266574    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:03:58.267536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:03:58.267519    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:04:09.562894 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:09.562855    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:04:11.245742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.245708    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"]
Nov 03 12:04:11.246270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.245914    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs" podUID="74160872-48f5-47de-94fe-01b5ba2afa8e" containerName="maas-api" containerID="cri-o://0b771e2efec673c66c259e0da3b4c3e88aa8f37d64e6bf9c253f9ec8e1c52198" gracePeriod=30
Nov 03 12:04:11.474041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.473987    4909 generic.go:356] "Generic (PLEG): container finished" podID="74160872-48f5-47de-94fe-01b5ba2afa8e" containerID="0b771e2efec673c66c259e0da3b4c3e88aa8f37d64e6bf9c253f9ec8e1c52198" exitCode=0
Nov 03 12:04:11.474260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.474049    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs" event={"ID":"74160872-48f5-47de-94fe-01b5ba2afa8e","Type":"ContainerDied","Data":"0b771e2efec673c66c259e0da3b4c3e88aa8f37d64e6bf9c253f9ec8e1c52198"}
Nov 03 12:04:11.524908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.524885    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:04:11.666236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.666209    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls\") pod \"74160872-48f5-47de-94fe-01b5ba2afa8e\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") "
Nov 03 12:04:11.666392 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.666243    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74160872-48f5-47de-94fe-01b5ba2afa8e-kube-api-access-x7d4h\" (UniqueName: \"kubernetes.io/projected/74160872-48f5-47de-94fe-01b5ba2afa8e-kube-api-access-x7d4h\") pod \"74160872-48f5-47de-94fe-01b5ba2afa8e\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") "
Nov 03 12:04:11.666392 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.666286    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-metrics-certs\") pod \"74160872-48f5-47de-94fe-01b5ba2afa8e\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") "
Nov 03 12:04:11.666392 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.666309    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74160872-48f5-47de-94fe-01b5ba2afa8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74160872-48f5-47de-94fe-01b5ba2afa8e-openshift-service-ca\") pod \"74160872-48f5-47de-94fe-01b5ba2afa8e\" (UID: \"74160872-48f5-47de-94fe-01b5ba2afa8e\") "
Nov 03 12:04:11.666793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.666757    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74160872-48f5-47de-94fe-01b5ba2afa8e-openshift-service-ca" pod "74160872-48f5-47de-94fe-01b5ba2afa8e" (UID: "74160872-48f5-47de-94fe-01b5ba2afa8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:11.668503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.668477    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls" pod "74160872-48f5-47de-94fe-01b5ba2afa8e" (UID: "74160872-48f5-47de-94fe-01b5ba2afa8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:11.668631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.668512    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74160872-48f5-47de-94fe-01b5ba2afa8e-kube-api-access-x7d4h" pod "74160872-48f5-47de-94fe-01b5ba2afa8e" (UID: "74160872-48f5-47de-94fe-01b5ba2afa8e"). InnerVolumeSpecName "kube-api-access-x7d4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:11.668631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.668584    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-metrics-certs" pod "74160872-48f5-47de-94fe-01b5ba2afa8e" (UID: "74160872-48f5-47de-94fe-01b5ba2afa8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:11.769779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.769521    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7d4h\" (UniqueName: \"kubernetes.io/projected/74160872-48f5-47de-94fe-01b5ba2afa8e-kube-api-access-x7d4h\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:11.769779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.769558    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:11.769779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.769573    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74160872-48f5-47de-94fe-01b5ba2afa8e-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:11.769779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:11.769583    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74160872-48f5-47de-94fe-01b5ba2afa8e-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:12.323459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.323425    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"]
Nov 03 12:04:12.323926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.323626    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8" podUID="094b2f92-4824-43ca-92dd-400c1d57ce3b" containerName="maas-api" containerID="cri-o://df421900cd5263ba4f7274c35cbceb3f02159c018a984f20dba385e1c8c8e372" gracePeriod=30
Nov 03 12:04:12.480657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.480628    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"
Nov 03 12:04:12.480859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.480663    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs" event={"ID":"74160872-48f5-47de-94fe-01b5ba2afa8e","Type":"ContainerDied","Data":"e3a987bffb89d91d84cc8122110e0c38ddfd440967328fb155d7af0a5f709888"}
Nov 03 12:04:12.480859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.480747    4909 scope.go:122] "RemoveContainer" containerID="0b771e2efec673c66c259e0da3b4c3e88aa8f37d64e6bf9c253f9ec8e1c52198"
Nov 03 12:04:12.482478 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.482453    4909 generic.go:356] "Generic (PLEG): container finished" podID="094b2f92-4824-43ca-92dd-400c1d57ce3b" containerID="df421900cd5263ba4f7274c35cbceb3f02159c018a984f20dba385e1c8c8e372" exitCode=0
Nov 03 12:04:12.482574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.482495    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8" event={"ID":"094b2f92-4824-43ca-92dd-400c1d57ce3b","Type":"ContainerDied","Data":"df421900cd5263ba4f7274c35cbceb3f02159c018a984f20dba385e1c8c8e372"}
Nov 03 12:04:12.581985 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.581920    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:04:12.631470 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.631445    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"]
Nov 03 12:04:12.635402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.635377    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f732077f-5b95797745-qs6hs"]
Nov 03 12:04:12.781182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.781141    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094b2f92-4824-43ca-92dd-400c1d57ce3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094b2f92-4824-43ca-92dd-400c1d57ce3b-openshift-service-ca\") pod \"094b2f92-4824-43ca-92dd-400c1d57ce3b\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") "
Nov 03 12:04:12.781731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.781660    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094b2f92-4824-43ca-92dd-400c1d57ce3b-openshift-service-ca" pod "094b2f92-4824-43ca-92dd-400c1d57ce3b" (UID: "094b2f92-4824-43ca-92dd-400c1d57ce3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:12.781838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.781811    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls\") pod \"094b2f92-4824-43ca-92dd-400c1d57ce3b\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") "
Nov 03 12:04:12.781899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.781866    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-metrics-certs\") pod \"094b2f92-4824-43ca-92dd-400c1d57ce3b\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") "
Nov 03 12:04:12.781950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.781911    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094b2f92-4824-43ca-92dd-400c1d57ce3b-kube-api-access-pc72j\" (UniqueName: \"kubernetes.io/projected/094b2f92-4824-43ca-92dd-400c1d57ce3b-kube-api-access-pc72j\") pod \"094b2f92-4824-43ca-92dd-400c1d57ce3b\" (UID: \"094b2f92-4824-43ca-92dd-400c1d57ce3b\") "
Nov 03 12:04:12.782267 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.782247    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094b2f92-4824-43ca-92dd-400c1d57ce3b-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:12.789114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.789080    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094b2f92-4824-43ca-92dd-400c1d57ce3b-kube-api-access-pc72j" pod "094b2f92-4824-43ca-92dd-400c1d57ce3b" (UID: "094b2f92-4824-43ca-92dd-400c1d57ce3b"). InnerVolumeSpecName "kube-api-access-pc72j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:12.790403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.790376    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-metrics-certs" pod "094b2f92-4824-43ca-92dd-400c1d57ce3b" (UID: "094b2f92-4824-43ca-92dd-400c1d57ce3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:12.790489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.790448    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls" pod "094b2f92-4824-43ca-92dd-400c1d57ce3b" (UID: "094b2f92-4824-43ca-92dd-400c1d57ce3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:12.883649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.883594    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc72j\" (UniqueName: \"kubernetes.io/projected/094b2f92-4824-43ca-92dd-400c1d57ce3b-kube-api-access-pc72j\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:12.883649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.883620    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:12.883649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:12.883630    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094b2f92-4824-43ca-92dd-400c1d57ce3b-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:13.121793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:13.121760    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74160872-48f5-47de-94fe-01b5ba2afa8e" path="/var/lib/kubelet/pods/74160872-48f5-47de-94fe-01b5ba2afa8e/volumes"
Nov 03 12:04:13.488986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:13.488952    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8" event={"ID":"094b2f92-4824-43ca-92dd-400c1d57ce3b","Type":"ContainerDied","Data":"8697c7b9238b01433f61c5385c737577bcca0a01b40f4a44de7555648da314ad"}
Nov 03 12:04:13.488986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:13.488977    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"
Nov 03 12:04:13.489498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:13.489024    4909 scope.go:122] "RemoveContainer" containerID="df421900cd5263ba4f7274c35cbceb3f02159c018a984f20dba385e1c8c8e372"
Nov 03 12:04:13.640062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:13.640033    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"]
Nov 03 12:04:13.646908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:13.646882    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b9370d1-6c66487bdf-fsjn8"]
Nov 03 12:04:15.012442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.012412    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"]
Nov 03 12:04:15.012889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.012849    4909 cpu_manager.go:414] "containerMap: removing container" podUID="094b2f92-4824-43ca-92dd-400c1d57ce3b" containerName="maas-api"
Nov 03 12:04:15.012889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.012860    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094b2f92-4824-43ca-92dd-400c1d57ce3b" containerName="maas-api"
Nov 03 12:04:15.012889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.012878    4909 cpu_manager.go:414] "containerMap: removing container" podUID="74160872-48f5-47de-94fe-01b5ba2afa8e" containerName="maas-api"
Nov 03 12:04:15.012889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.012883    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74160872-48f5-47de-94fe-01b5ba2afa8e" containerName="maas-api"
Nov 03 12:04:15.013039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.012979    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="094b2f92-4824-43ca-92dd-400c1d57ce3b" containerName="maas-api"
Nov 03 12:04:15.013039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.012992    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="74160872-48f5-47de-94fe-01b5ba2afa8e" containerName="maas-api"
Nov 03 12:04:15.016455 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.016436    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.072721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.066424    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91cd99-openshift-default-dockercfg-pj65x\""
Nov 03 12:04:15.100776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.100745    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.100776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.100780    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fdt\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-kube-api-access-v2fdt\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.101008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.100803    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.101008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.100949    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.101080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.101032    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.101080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.101067    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.101142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.101128    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.101187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.101167    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.101224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.101199    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.113625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.113603    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"]
Nov 03 12:04:15.121869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.121847    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094b2f92-4824-43ca-92dd-400c1d57ce3b" path="/var/lib/kubelet/pods/094b2f92-4824-43ca-92dd-400c1d57ce3b/volumes"
Nov 03 12:04:15.201689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.201658    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.201840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.201708    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.201840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.201818    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.201972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.201861    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.201972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.201896    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.201972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.201947    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.201975    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fdt\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-kube-api-access-v2fdt\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.202006    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.202028    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.202048    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.202414    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202575 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.202420    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.202758    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.202908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.202766    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.204744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.204693    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.205079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.205059    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.272509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.271728    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.273289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.273257    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fdt\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-kube-api-access-v2fdt\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.327563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.327529    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:15.738751 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.738728    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"]
Nov 03 12:04:15.740328 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:04:15.740296    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2fdd5a_fb58_4278_89bc_3f79da765f56.slice/crio-6b8acb75e66771c1707ad78f5f01594050afa45fb688a4caadfea7ef6cea07b1 WatchSource:0}: Error finding container 6b8acb75e66771c1707ad78f5f01594050afa45fb688a4caadfea7ef6cea07b1: Status 404 returned error can't find the container with id 6b8acb75e66771c1707ad78f5f01594050afa45fb688a4caadfea7ef6cea07b1
Nov 03 12:04:15.742640 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.742610    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:15.742739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.742688    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:15.742739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:15.742733    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:16.451305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:16.451266    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"]
Nov 03 12:04:16.504785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:16.504754    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" event={"ID":"ec2fdd5a-fb58-4278-89bc-3f79da765f56","Type":"ContainerStarted","Data":"009f55c1050061fecfb26d7e183dc3b8424e026afe3f0e6497a222e2b3dc7cfc"}
Nov 03 12:04:16.504940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:16.504803    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" event={"ID":"ec2fdd5a-fb58-4278-89bc-3f79da765f56","Type":"ContainerStarted","Data":"6b8acb75e66771c1707ad78f5f01594050afa45fb688a4caadfea7ef6cea07b1"}
Nov 03 12:04:16.665859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:16.665811    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" podStartSLOduration=2.665798073 podStartE2EDuration="2.665798073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:16.664005683 +0000 UTC m=+805.676472345" watchObservedRunningTime="2026-11-03 12:04:16.665798073 +0000 UTC m=+805.678264804"
Nov 03 12:04:17.328492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:17.328460    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:17.333493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:17.333466    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:17.509555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:17.509525    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:17.510377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:17.510362    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:18.287609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:18.287571    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:04:21.452007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.451969    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j" podUID="60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" containerName="payload-pre-processing" containerID="cri-o://a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7" gracePeriod=25
Nov 03 12:04:21.701462 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.701437    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:04:21.763056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.762986    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-kube-api-access-g5hgk\" (UniqueName: \"kubernetes.io/projected/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-kube-api-access-g5hgk\") pod \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") "
Nov 03 12:04:21.763181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.763089    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-plugins-config-volume\") pod \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\" (UID: \"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4\") "
Nov 03 12:04:21.763464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.763438    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-plugins-config-volume" pod "60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" (UID: "60616d0a-9b4c-473d-9ec9-1ae2efff8fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:21.765206 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.765186    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-kube-api-access-g5hgk" pod "60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" (UID: "60616d0a-9b4c-473d-9ec9-1ae2efff8fa4"). InnerVolumeSpecName "kube-api-access-g5hgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:21.864159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.864132    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:21.864299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:21.864167    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hgk\" (UniqueName: \"kubernetes.io/projected/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4-kube-api-access-g5hgk\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:22.533044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.533007    4909 generic.go:356] "Generic (PLEG): container finished" podID="60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" containerID="a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7" exitCode=0
Nov 03 12:04:22.533530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.533049    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"
Nov 03 12:04:22.533530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.533095    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j" event={"ID":"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4","Type":"ContainerDied","Data":"a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7"}
Nov 03 12:04:22.533530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.533161    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j" event={"ID":"60616d0a-9b4c-473d-9ec9-1ae2efff8fa4","Type":"ContainerDied","Data":"be6bbbee2cc934727dec7664e1a931e881f7dcbe1d5361bbc8e1ec641b5dd4b8"}
Nov 03 12:04:22.533530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.533195    4909 scope.go:122] "RemoveContainer" containerID="a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7"
Nov 03 12:04:22.542346 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.542331    4909 scope.go:122] "RemoveContainer" containerID="a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7"
Nov 03 12:04:22.542564 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:22.542546    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7\": container with ID starting with a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7 not found: ID does not exist" containerID="a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7"
Nov 03 12:04:22.542624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.542577    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7"} err="failed to get container status \"a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7\": rpc error: code = NotFound desc = could not find container \"a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7\": container with ID starting with a92c3a9d851d92d1b73cfd0fe75d5cbc5e125214e9bd370826b2e880292484e7 not found: ID does not exist"
Nov 03 12:04:22.684076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.684048    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"]
Nov 03 12:04:22.730726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:22.730672    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-30e6ec98-7b8fdb7cbc-wnb8j"]
Nov 03 12:04:23.102454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.102417    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"]
Nov 03 12:04:23.102650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.102626    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" podUID="29c3188f-a6a0-4215-beb4-d89e60c9b0fa" containerName="maas-api" containerID="cri-o://32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03" gracePeriod=30
Nov 03 12:04:23.124137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.124109    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" path="/var/lib/kubelet/pods/60616d0a-9b4c-473d-9ec9-1ae2efff8fa4/volumes"
Nov 03 12:04:23.208087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.208046    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" podUID="29c3188f-a6a0-4215-beb4-d89e60c9b0fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.46:8443/health\": dial tcp 10.132.0.46:8443: connect: connection refused"
Nov 03 12:04:23.384225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.384203    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:04:23.485424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.485389    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-metrics-certs\") pod \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") "
Nov 03 12:04:23.485594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.485441    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls\") pod \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") "
Nov 03 12:04:23.485594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.485481    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-kube-api-access-h4cv2\" (UniqueName: \"kubernetes.io/projected/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-kube-api-access-h4cv2\") pod \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") "
Nov 03 12:04:23.485594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.485507    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-openshift-service-ca\") pod \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\" (UID: \"29c3188f-a6a0-4215-beb4-d89e60c9b0fa\") "
Nov 03 12:04:23.485939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.485914    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-openshift-service-ca" pod "29c3188f-a6a0-4215-beb4-d89e60c9b0fa" (UID: "29c3188f-a6a0-4215-beb4-d89e60c9b0fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:23.487630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.487605    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls" pod "29c3188f-a6a0-4215-beb4-d89e60c9b0fa" (UID: "29c3188f-a6a0-4215-beb4-d89e60c9b0fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:23.487749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.487658    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-kube-api-access-h4cv2" pod "29c3188f-a6a0-4215-beb4-d89e60c9b0fa" (UID: "29c3188f-a6a0-4215-beb4-d89e60c9b0fa"). InnerVolumeSpecName "kube-api-access-h4cv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:23.487749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.487721    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-metrics-certs" pod "29c3188f-a6a0-4215-beb4-d89e60c9b0fa" (UID: "29c3188f-a6a0-4215-beb4-d89e60c9b0fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:23.545346 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.545312    4909 generic.go:356] "Generic (PLEG): container finished" podID="29c3188f-a6a0-4215-beb4-d89e60c9b0fa" containerID="32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03" exitCode=0
Nov 03 12:04:23.545735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.545361    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"
Nov 03 12:04:23.545735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.545394    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" event={"ID":"29c3188f-a6a0-4215-beb4-d89e60c9b0fa","Type":"ContainerDied","Data":"32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03"}
Nov 03 12:04:23.545735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.545461    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65" event={"ID":"29c3188f-a6a0-4215-beb4-d89e60c9b0fa","Type":"ContainerDied","Data":"4ab7a2694761538d27556e634205b0b6d31840b4471b073d8158484f077b2b42"}
Nov 03 12:04:23.545735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.545490    4909 scope.go:122] "RemoveContainer" containerID="32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03"
Nov 03 12:04:23.555182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.555166    4909 scope.go:122] "RemoveContainer" containerID="32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03"
Nov 03 12:04:23.555460 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:23.555442    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03\": container with ID starting with 32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03 not found: ID does not exist" containerID="32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03"
Nov 03 12:04:23.555523 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.555467    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03"} err="failed to get container status \"32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03\": rpc error: code = NotFound desc = could not find container \"32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03\": container with ID starting with 32cf1ce8aea934dafbb46c7753718a6a9f7c36677a58cc314fa141a581f14d03 not found: ID does not exist"
Nov 03 12:04:23.587145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.587123    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:23.587145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.587144    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:23.587273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.587153    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4cv2\" (UniqueName: \"kubernetes.io/projected/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-kube-api-access-h4cv2\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:23.587273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.587162    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c3188f-a6a0-4215-beb4-d89e60c9b0fa-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:23.697406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.697379    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"]
Nov 03 12:04:23.701173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:23.701147    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1319e7ef-7f56db9866-tpd65"]
Nov 03 12:04:25.122272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:25.122228    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c3188f-a6a0-4215-beb4-d89e60c9b0fa" path="/var/lib/kubelet/pods/29c3188f-a6a0-4215-beb4-d89e60c9b0fa/volumes"
Nov 03 12:04:28.090994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:28.090965    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:04:29.747658 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:29.747627    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"]
Nov 03 12:04:29.748096 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:29.747824    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7" podUID="947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" containerName="maas-api" containerID="cri-o://3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc" gracePeriod=30
Nov 03 12:04:30.025490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.025463    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:04:30.162939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.162912    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-openshift-service-ca\") pod \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") "
Nov 03 12:04:30.163119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.162966    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls\") pod \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") "
Nov 03 12:04:30.163119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.163003    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-metrics-certs\") pod \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") "
Nov 03 12:04:30.163119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.163058    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-kube-api-access-7rllj\" (UniqueName: \"kubernetes.io/projected/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-kube-api-access-7rllj\") pod \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\" (UID: \"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8\") "
Nov 03 12:04:30.163360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.163330    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-openshift-service-ca" pod "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" (UID: "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:30.165190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.165166    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-metrics-certs" pod "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" (UID: "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:30.165299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.165202    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls" pod "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" (UID: "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:30.165299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.165279    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-kube-api-access-7rllj" pod "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" (UID: "947f99e7-60d5-45bf-8ffa-b2a34b30a5c8"). InnerVolumeSpecName "kube-api-access-7rllj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:30.264376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.264312    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:30.264376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.264338    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:30.264376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.264348    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:30.264376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.264357    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rllj\" (UniqueName: \"kubernetes.io/projected/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8-kube-api-access-7rllj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:30.581022 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.580940    4909 generic.go:356] "Generic (PLEG): container finished" podID="947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" containerID="3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc" exitCode=0
Nov 03 12:04:30.581022 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.580999    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"
Nov 03 12:04:30.581254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.580989    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7" event={"ID":"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8","Type":"ContainerDied","Data":"3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc"}
Nov 03 12:04:30.581254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.581135    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7" event={"ID":"947f99e7-60d5-45bf-8ffa-b2a34b30a5c8","Type":"ContainerDied","Data":"27414e6474a6a9cfe9cbad40213fc2d0b133b256e6d24cb9b7c097a5bb514dd9"}
Nov 03 12:04:30.581254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.581162    4909 scope.go:122] "RemoveContainer" containerID="3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc"
Nov 03 12:04:30.590459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.590442    4909 scope.go:122] "RemoveContainer" containerID="3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc"
Nov 03 12:04:30.590838 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:30.590739    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc\": container with ID starting with 3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc not found: ID does not exist" containerID="3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc"
Nov 03 12:04:30.590838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.590768    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc"} err="failed to get container status \"3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc\": rpc error: code = NotFound desc = could not find container \"3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc\": container with ID starting with 3dde2650685d992856a0e279426d0acc3c586e876618103f50a9a74c6438e5bc not found: ID does not exist"
Nov 03 12:04:30.733681 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.733643    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"]
Nov 03 12:04:30.735000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:30.734977    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e6ec98-58d777b6fc-gfbf7"]
Nov 03 12:04:31.121998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.121965    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" path="/var/lib/kubelet/pods/947f99e7-60d5-45bf-8ffa-b2a34b30a5c8/volumes"
Nov 03 12:04:31.409987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.409912    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"]
Nov 03 12:04:31.410355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410339    4909 cpu_manager.go:414] "containerMap: removing container" podUID="60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" containerName="payload-pre-processing"
Nov 03 12:04:31.410415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410357    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" containerName="payload-pre-processing"
Nov 03 12:04:31.410415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410394    4909 cpu_manager.go:414] "containerMap: removing container" podUID="29c3188f-a6a0-4215-beb4-d89e60c9b0fa" containerName="maas-api"
Nov 03 12:04:31.410415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410400    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c3188f-a6a0-4215-beb4-d89e60c9b0fa" containerName="maas-api"
Nov 03 12:04:31.410415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410415    4909 cpu_manager.go:414] "containerMap: removing container" podUID="947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" containerName="maas-api"
Nov 03 12:04:31.410588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410421    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" containerName="maas-api"
Nov 03 12:04:31.410588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410495    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c3188f-a6a0-4215-beb4-d89e60c9b0fa" containerName="maas-api"
Nov 03 12:04:31.410588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410508    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="947f99e7-60d5-45bf-8ffa-b2a34b30a5c8" containerName="maas-api"
Nov 03 12:04:31.410588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.410515    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="60616d0a-9b4c-473d-9ec9-1ae2efff8fa4" containerName="payload-pre-processing"
Nov 03 12:04:31.413939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.413917    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.462231 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.462209    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-91cd99-serving-cert\""
Nov 03 12:04:31.477962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.477931    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.478072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.477980    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-metrics-certs\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.478072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.478050    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11e72b2-5094-45a0-b147-7d86098e89c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.478213 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.478193    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjkz\" (UniqueName: \"kubernetes.io/projected/d11e72b2-5094-45a0-b147-7d86098e89c3-kube-api-access-qzjkz\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.510029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.510003    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"]
Nov 03 12:04:31.579513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.579485    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjkz\" (UniqueName: \"kubernetes.io/projected/d11e72b2-5094-45a0-b147-7d86098e89c3-kube-api-access-qzjkz\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.579660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.579534    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.579660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.579565    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-metrics-certs\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.579660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.579593    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11e72b2-5094-45a0-b147-7d86098e89c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.579845 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:31.579708    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-serving-cert: secret "maas-api-e2e-worker-w2-91cd99-serving-cert" not found
Nov 03 12:04:31.579845 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:31.579780    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls podName:d11e72b2-5094-45a0-b147-7d86098e89c3 nodeName:}" failed. No retries permitted until 2026-11-03 12:04:32.07976579 +0000 UTC m=+821.092232437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls") pod "maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" (UID: "d11e72b2-5094-45a0-b147-7d86098e89c3") : secret "maas-api-e2e-worker-w2-91cd99-serving-cert" not found
Nov 03 12:04:31.580320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.580302    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11e72b2-5094-45a0-b147-7d86098e89c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.582229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.582208    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-metrics-certs\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:31.637161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.637130    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"]
Nov 03 12:04:31.637344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.637320    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy" containerID="cri-o://8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed" gracePeriod=30
Nov 03 12:04:31.641728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:31.639071    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjkz\" (UniqueName: \"kubernetes.io/projected/d11e72b2-5094-45a0-b147-7d86098e89c3-kube-api-access-qzjkz\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:32.082680 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:32.082645    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:32.084988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:32.084966    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:32.330502 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:32.330465    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:32.695284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:32.693562    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"]
Nov 03 12:04:32.702040 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:04:32.701996    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11e72b2_5094_45a0_b147_7d86098e89c3.slice/crio-da9b659043148edbd5b8943dbee019026da9c06b36a688c52bfab0f788372ff9 WatchSource:0}: Error finding container da9b659043148edbd5b8943dbee019026da9c06b36a688c52bfab0f788372ff9: Status 404 returned error can't find the container with id da9b659043148edbd5b8943dbee019026da9c06b36a688c52bfab0f788372ff9
Nov 03 12:04:32.972062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:32.971990    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"]
Nov 03 12:04:32.972245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:32.972204    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" containerID="cri-o://423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42" gracePeriod=30
Nov 03 12:04:33.298927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.298856    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"]
Nov 03 12:04:33.302474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.302454    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.351015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.350988    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9893f0-openshift-default-dockercfg-ptlmk\""
Nov 03 12:04:33.361300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.361272    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"]
Nov 03 12:04:33.396121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396094    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396137    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d9fb72-0575-4a4c-a47d-fd875216453c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396156    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396222    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396272    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396290    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396330    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396347    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.396519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.396432    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmmm\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-kube-api-access-sbmmm\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.496990 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.496962    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.496994    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497014    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497030    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497131    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmmm\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-kube-api-access-sbmmm\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497207    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497254    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d9fb72-0575-4a4c-a47d-fd875216453c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497279    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497302    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497417    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497498    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497663 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497585    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.497663 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.497618    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.498246 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.498226    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d9fb72-0575-4a4c-a47d-fd875216453c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.499613 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.499583    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.499741 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.499618    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.551007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.550881    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.551134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.551119    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmmm\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-kube-api-access-sbmmm\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:33.597896 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.597860    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" event={"ID":"d11e72b2-5094-45a0-b147-7d86098e89c3","Type":"ContainerStarted","Data":"da9b659043148edbd5b8943dbee019026da9c06b36a688c52bfab0f788372ff9"}
Nov 03 12:04:33.614549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:33.614515    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:34.026555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.026529    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"]
Nov 03 12:04:34.028663 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:04:34.028632    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d9fb72_0575_4a4c_a47d_fd875216453c.slice/crio-39abe6b39a4509f414ae748e86a857b63b362bea2117c6dcddb03ef6db42f52e WatchSource:0}: Error finding container 39abe6b39a4509f414ae748e86a857b63b362bea2117c6dcddb03ef6db42f52e: Status 404 returned error can't find the container with id 39abe6b39a4509f414ae748e86a857b63b362bea2117c6dcddb03ef6db42f52e
Nov 03 12:04:34.031484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.031452    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:34.031584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.031529    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:34.031584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.031576    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:34.126855 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.126822    4909 patch_prober.go:29] interesting pod/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:04:34.127015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.126893    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:04:34.604869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.604837    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" event={"ID":"d11e72b2-5094-45a0-b147-7d86098e89c3","Type":"ContainerStarted","Data":"fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec"}
Nov 03 12:04:34.605359 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.604951    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:34.606561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.606532    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" event={"ID":"12d9fb72-0575-4a4c-a47d-fd875216453c","Type":"ContainerStarted","Data":"c3b45b148eedde5319edd61885d7e8541b20818353e73bbadbc5242c111d136b"}
Nov 03 12:04:34.606689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.606573    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" event={"ID":"12d9fb72-0575-4a4c-a47d-fd875216453c","Type":"ContainerStarted","Data":"39abe6b39a4509f414ae748e86a857b63b362bea2117c6dcddb03ef6db42f52e"}
Nov 03 12:04:34.615018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.614995    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:34.711884 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:34.711829    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" podStartSLOduration=2.813027751 podStartE2EDuration="3.711812862s" totalImagesPullingTime="898.785111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:34.710880856 +0000 UTC m=+823.723347518" watchObservedRunningTime="2026-11-03 12:04:34.711812862 +0000 UTC m=+823.724279526"
Nov 03 12:04:35.621015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:35.620987    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:35.734026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:35.733974    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" podStartSLOduration=2.7339599999999997 podStartE2EDuration="2.73396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:34.834021145 +0000 UTC m=+823.846487832" watchObservedRunningTime="2026-11-03 12:04:35.73396 +0000 UTC m=+824.746426700"
Nov 03 12:04:36.616158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:36.616131    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:36.617143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:36.617124    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:04:36.894321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:36.894298    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:04:37.040454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.040406    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-credential-socket\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.040661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.040471    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-token\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-token\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.040661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.040520    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-kube-api-access-fn4xz\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-kube-api-access-fn4xz\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.040820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.040777    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de78094-d888-4fc0-8a19-f781d7a898ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de78094-d888-4fc0-8a19-f781d7a898ac-istiod-ca-cert\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.040879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.040849    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-data\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.041121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.040934    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-envoy\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.041121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.040977    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-podinfo\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.041121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.041015    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-certs\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.041121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.041056    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-socket\") pod \"4de78094-d888-4fc0-8a19-f781d7a898ac\" (UID: \"4de78094-d888-4fc0-8a19-f781d7a898ac\") "
Nov 03 12:04:37.042101 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.042071    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-socket" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.042339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.042320    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-credential-socket" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.047065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.047045    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-data" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.055198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.055172    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de78094-d888-4fc0-8a19-f781d7a898ac-istiod-ca-cert" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:37.055553 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.055468    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-certs" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.060860 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.060835    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-kube-api-access-fn4xz" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "kube-api-access-fn4xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:37.065843 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.065813    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-token" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:37.066405 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.066377    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-podinfo" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:04:37.068841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.068817    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-envoy" pod "4de78094-d888-4fc0-8a19-f781d7a898ac" (UID: "4de78094-d888-4fc0-8a19-f781d7a898ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.142550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142526    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142549    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142558    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4xz\" (UniqueName: \"kubernetes.io/projected/4de78094-d888-4fc0-8a19-f781d7a898ac-kube-api-access-fn4xz\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142568    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de78094-d888-4fc0-8a19-f781d7a898ac-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142576    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142584    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142592    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de78094-d888-4fc0-8a19-f781d7a898ac-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142602    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.142679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.142610    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de78094-d888-4fc0-8a19-f781d7a898ac-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.318891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.318860    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:04:37.622027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.621991    4909 generic.go:356] "Generic (PLEG): container finished" podID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerID="8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed" exitCode=0
Nov 03 12:04:37.622198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.622040    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"
Nov 03 12:04:37.622198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.622075    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" event={"ID":"4de78094-d888-4fc0-8a19-f781d7a898ac","Type":"ContainerDied","Data":"8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed"}
Nov 03 12:04:37.622198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.622131    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j" event={"ID":"4de78094-d888-4fc0-8a19-f781d7a898ac","Type":"ContainerDied","Data":"0ead8414d2bbd610d1512d8149215b309d81afd3a24efcceee19e573dd512439"}
Nov 03 12:04:37.622198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.622154    4909 scope.go:122] "RemoveContainer" containerID="8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed"
Nov 03 12:04:37.631500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.631477    4909 scope.go:122] "RemoveContainer" containerID="ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9"
Nov 03 12:04:37.664010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.663453    4909 scope.go:122] "RemoveContainer" containerID="8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed"
Nov 03 12:04:37.664010 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:37.663948    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed\": container with ID starting with 8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed not found: ID does not exist" containerID="8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed"
Nov 03 12:04:37.664010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.663983    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed"} err="failed to get container status \"8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed\": rpc error: code = NotFound desc = could not find container \"8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed\": container with ID starting with 8862123b889c07e1ba09a0e411db642cc92a60069613402daccaedc637a5e9ed not found: ID does not exist"
Nov 03 12:04:37.664010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.664003    4909 scope.go:122] "RemoveContainer" containerID="ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9"
Nov 03 12:04:37.664425 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:37.664324    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9\": container with ID starting with ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9 not found: ID does not exist" containerID="ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9"
Nov 03 12:04:37.664425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.664363    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9"} err="failed to get container status \"ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9\": rpc error: code = NotFound desc = could not find container \"ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9\": container with ID starting with ad6007d9b4b38dd393a7b79ebd6802e80b42690190e098ba87070c5374abf1b9 not found: ID does not exist"
Nov 03 12:04:37.780052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.780018    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"]
Nov 03 12:04:37.825562 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:37.825536    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1319e7ef-openshift-default-6575b7cd87-rz97j"]
Nov 03 12:04:38.275248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.275217    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:04:38.369749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.369692    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-credential-socket\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.369944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.369766    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-kube-api-access-6cn77\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-kube-api-access-6cn77\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.369944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.369837    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-podinfo\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.369944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.369865    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istiod-ca-cert\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.369944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.369910    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-token\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-token\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.370109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.369950    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-certs\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.370109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.369991    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-credential-socket" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:38.370109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370010    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-socket\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.370109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370057    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-envoy\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.370262 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370124    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-data\") pod \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\" (UID: \"f806bab1-f23a-4ca3-865f-ea5b34aa2d49\") "
Nov 03 12:04:38.370262 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370221    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istiod-ca-cert" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:38.370654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370495    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.370654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370522    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.370654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370607    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-socket" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:38.370935 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370672    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-certs" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:38.370935 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.370683    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-data" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:38.372592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.372557    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-token" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:38.372844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.372824    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-kube-api-access-6cn77" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "kube-api-access-6cn77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:38.373019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.372991    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-envoy" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:38.373413 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.373397    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-podinfo" pod "f806bab1-f23a-4ca3-865f-ea5b34aa2d49" (UID: "f806bab1-f23a-4ca3-865f-ea5b34aa2d49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:04:38.471669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.471633    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cn77\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-kube-api-access-6cn77\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.471669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.471660    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.471669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.471670    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.471669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.471680    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.471983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.471689    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.471983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.471723    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.471983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.471732    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f806bab1-f23a-4ca3-865f-ea5b34aa2d49-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:38.629874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.629789    4909 generic.go:356] "Generic (PLEG): container finished" podID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerID="423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42" exitCode=0
Nov 03 12:04:38.629874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.629840    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"
Nov 03 12:04:38.630067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.629875    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" event={"ID":"f806bab1-f23a-4ca3-865f-ea5b34aa2d49","Type":"ContainerDied","Data":"423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42"}
Nov 03 12:04:38.630067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.629926    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw" event={"ID":"f806bab1-f23a-4ca3-865f-ea5b34aa2d49","Type":"ContainerDied","Data":"7056b6153acc496c3b83bbb11defa7581b9c14e2cb796c6f25b32a96061c1d14"}
Nov 03 12:04:38.630067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.629949    4909 scope.go:122] "RemoveContainer" containerID="423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42"
Nov 03 12:04:38.667847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.667820    4909 scope.go:122] "RemoveContainer" containerID="208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960"
Nov 03 12:04:38.677679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.677658    4909 scope.go:122] "RemoveContainer" containerID="423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42"
Nov 03 12:04:38.678001 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:38.677973    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42\": container with ID starting with 423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42 not found: ID does not exist" containerID="423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42"
Nov 03 12:04:38.678080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.678010    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42"} err="failed to get container status \"423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42\": rpc error: code = NotFound desc = could not find container \"423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42\": container with ID starting with 423b4c350961fb6c51688b958317173b1270a1881c98bd447932a50f7a225e42 not found: ID does not exist"
Nov 03 12:04:38.678080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.678029    4909 scope.go:122] "RemoveContainer" containerID="208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960"
Nov 03 12:04:38.678273 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:38.678258    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960\": container with ID starting with 208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960 not found: ID does not exist" containerID="208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960"
Nov 03 12:04:38.678318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.678279    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960"} err="failed to get container status \"208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960\": rpc error: code = NotFound desc = could not find container \"208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960\": container with ID starting with 208a5055526160b2fe0515b8aa10e0ff547f97ca3b4d01c9a50f5f7c6a74d960 not found: ID does not exist"
Nov 03 12:04:38.781925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.781897    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"]
Nov 03 12:04:38.827652 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:38.827624    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-30e6ec98-gw-openshift-default-5cb78bfd6f-lhqtw"]
Nov 03 12:04:39.122324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:39.122294    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" path="/var/lib/kubelet/pods/4de78094-d888-4fc0-8a19-f781d7a898ac/volumes"
Nov 03 12:04:39.122775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:39.122760    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" path="/var/lib/kubelet/pods/f806bab1-f23a-4ca3-865f-ea5b34aa2d49/volumes"
Nov 03 12:04:39.587519 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:04:39.587488    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:04:40.565366 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.565333    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"]
Nov 03 12:04:40.565813 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.565793    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy"
Nov 03 12:04:40.565913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.565814    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy"
Nov 03 12:04:40.565913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.565837    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy"
Nov 03 12:04:40.565913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.565846    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy"
Nov 03 12:04:40.565913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.565901    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy"
Nov 03 12:04:40.565913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.565911    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy"
Nov 03 12:04:40.566105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.566012    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy"
Nov 03 12:04:40.566105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.566024    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy"
Nov 03 12:04:40.566105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.566039    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy"
Nov 03 12:04:40.569192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.569178    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.593932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.593901    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.594233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.594080    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8zp\" (UniqueName: \"kubernetes.io/projected/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-kube-api-access-dx8zp\") pod \"payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.627173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.627143    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 12:04:40.632151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.632118    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"]
Nov 03 12:04:40.668113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.667924    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8d2c7c\""
Nov 03 12:04:40.668113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.668026    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8d2c7c-dockercfg-j2zbg\""
Nov 03 12:04:40.695325 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.695266    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.696074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.696028    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.696223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.696203    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8zp\" (UniqueName: \"kubernetes.io/projected/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-kube-api-access-dx8zp\") pod \"payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.721439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.720089    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"]
Nov 03 12:04:40.721439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.720488    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy"
Nov 03 12:04:40.721439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.720499    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de78094-d888-4fc0-8a19-f781d7a898ac" containerName="istio-proxy"
Nov 03 12:04:40.721439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.720582    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f806bab1-f23a-4ca3-865f-ea5b34aa2d49" containerName="istio-proxy"
Nov 03 12:04:40.723007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.722725    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:40.782097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.774349    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8zp\" (UniqueName: \"kubernetes.io/projected/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-kube-api-access-dx8zp\") pod \"payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.790105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.790080    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"]
Nov 03 12:04:40.798229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.797730    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b7280-6313-425a-b48f-1a2f19666a74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:40.798229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.797847    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jx9k\" (UniqueName: \"kubernetes.io/projected/9b7b7280-6313-425a-b48f-1a2f19666a74-kube-api-access-9jx9k\") pod \"payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:40.886751 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.886650    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:40.898716 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.898675    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jx9k\" (UniqueName: \"kubernetes.io/projected/9b7b7280-6313-425a-b48f-1a2f19666a74-kube-api-access-9jx9k\") pod \"payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:40.898843 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.898821    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b7280-6313-425a-b48f-1a2f19666a74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:40.900753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.900723    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b7280-6313-425a-b48f-1a2f19666a74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:40.966444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:40.966404    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jx9k\" (UniqueName: \"kubernetes.io/projected/9b7b7280-6313-425a-b48f-1a2f19666a74-kube-api-access-9jx9k\") pod \"payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:41.049725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.037754    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:41.300266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.300232    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"]
Nov 03 12:04:41.303515 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:04:41.303486    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4c33dd_116d_4fbc_bfe7_d430bd3d7c58.slice/crio-07dc4ea0c66b1fecb08b38ac879de29e762db6862cb33655a89f6cfc9d61c945 WatchSource:0}: Error finding container 07dc4ea0c66b1fecb08b38ac879de29e762db6862cb33655a89f6cfc9d61c945: Status 404 returned error can't find the container with id 07dc4ea0c66b1fecb08b38ac879de29e762db6862cb33655a89f6cfc9d61c945
Nov 03 12:04:41.462741 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.462383    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"]
Nov 03 12:04:41.468079 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:04:41.468047    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7b7280_6313_425a_b48f_1a2f19666a74.slice/crio-95821de31f3aaf041527365357e359b1dcc914e9cae53f128cac220c46308319 WatchSource:0}: Error finding container 95821de31f3aaf041527365357e359b1dcc914e9cae53f128cac220c46308319: Status 404 returned error can't find the container with id 95821de31f3aaf041527365357e359b1dcc914e9cae53f128cac220c46308319
Nov 03 12:04:41.690271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.690235    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" event={"ID":"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58","Type":"ContainerStarted","Data":"fabd6a23b3f0c47513485afa3835cd70b507b32c99e03633e8a3e3d9996246a6"}
Nov 03 12:04:41.690832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.690291    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" event={"ID":"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58","Type":"ContainerStarted","Data":"07dc4ea0c66b1fecb08b38ac879de29e762db6862cb33655a89f6cfc9d61c945"}
Nov 03 12:04:41.690991 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.690963    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:41.692381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.692314    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" event={"ID":"9b7b7280-6313-425a-b48f-1a2f19666a74","Type":"ContainerStarted","Data":"749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe"}
Nov 03 12:04:41.692381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.692357    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" event={"ID":"9b7b7280-6313-425a-b48f-1a2f19666a74","Type":"ContainerStarted","Data":"95821de31f3aaf041527365357e359b1dcc914e9cae53f128cac220c46308319"}
Nov 03 12:04:41.692542 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.692481    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:41.808914 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:41.808865    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" podStartSLOduration=1.808851163 podStartE2EDuration="1.808851163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:41.807829955 +0000 UTC m=+830.820296618" watchObservedRunningTime="2026-11-03 12:04:41.808851163 +0000 UTC m=+830.821317804"
Nov 03 12:04:45.051038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.050976    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" podStartSLOduration=5.050956824 podStartE2EDuration="5.050956824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:41.920532163 +0000 UTC m=+830.932998827" watchObservedRunningTime="2026-11-03 12:04:45.050956824 +0000 UTC m=+834.063423489"
Nov 03 12:04:45.051525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.051232    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"]
Nov 03 12:04:45.054784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.054762    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.146411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.146342    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"]
Nov 03 12:04:45.153115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153080    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153138    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153224    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153324    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8272\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-kube-api-access-c8272\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153395    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153507 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153426    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153507 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153458    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153507 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153492    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.153664 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.153547    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.192923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.192892    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5951bc-openshift-default-dockercfg-bp9bz\""
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254343    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8272\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-kube-api-access-c8272\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254412    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254444    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254477    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254513    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254546    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254644    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254686    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.254727    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.255129    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.255177    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.255760    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.255956    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.256855 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.256567    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.258046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.258019    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.259246 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.259219    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.308176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.308111    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8272\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-kube-api-access-c8272\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.308362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.308342    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.367755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.367723    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:45.779343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.779318    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"]
Nov 03 12:04:45.781897 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:04:45.781865    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6815b8_75a6_4b9c_bc01_aac63d707d94.slice/crio-6914831d8433881ede31c0c3e4fd62155aefeeb1a46c8dfd8c97dc92909fbc44 WatchSource:0}: Error finding container 6914831d8433881ede31c0c3e4fd62155aefeeb1a46c8dfd8c97dc92909fbc44: Status 404 returned error can't find the container with id 6914831d8433881ede31c0c3e4fd62155aefeeb1a46c8dfd8c97dc92909fbc44
Nov 03 12:04:45.784251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.784222    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:45.784331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.784283    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:45.784331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:45.784313    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:46.044947 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.044857    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:04:46.728065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.728031    4909 generic.go:356] "Generic (PLEG): container finished" podID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerID="009f55c1050061fecfb26d7e183dc3b8424e026afe3f0e6497a222e2b3dc7cfc" exitCode=0
Nov 03 12:04:46.728521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.728103    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" event={"ID":"ec2fdd5a-fb58-4278-89bc-3f79da765f56","Type":"ContainerDied","Data":"009f55c1050061fecfb26d7e183dc3b8424e026afe3f0e6497a222e2b3dc7cfc"}
Nov 03 12:04:46.728521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.728498    4909 scope.go:122] "RemoveContainer" containerID="009f55c1050061fecfb26d7e183dc3b8424e026afe3f0e6497a222e2b3dc7cfc"
Nov 03 12:04:46.729975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.729933    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:46.730079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.729972    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" event={"ID":"4c6815b8-75a6-4b9c-bc01-aac63d707d94","Type":"ContainerStarted","Data":"cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b"}
Nov 03 12:04:46.730079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.730010    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" event={"ID":"4c6815b8-75a6-4b9c-bc01-aac63d707d94","Type":"ContainerStarted","Data":"6914831d8433881ede31c0c3e4fd62155aefeeb1a46c8dfd8c97dc92909fbc44"}
Nov 03 12:04:46.730079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.730020    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:46.730079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:46.730070    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:04:47.034609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:47.034507    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" podStartSLOduration=2.034492803 podStartE2EDuration="2.034492803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:47.031396852 +0000 UTC m=+836.043863534" watchObservedRunningTime="2026-11-03 12:04:47.034492803 +0000 UTC m=+836.046959465"
Nov 03 12:04:47.368164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:47.368061    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:47.373099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:47.373069    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:47.516879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:47.516844    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:47.736453 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:47.736413    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" event={"ID":"ec2fdd5a-fb58-4278-89bc-3f79da765f56","Type":"ContainerStarted","Data":"7105956ad7fb3fd5818134f840ca6c18c956f819d55fcf59484ff6335db51626"}
Nov 03 12:04:47.736884 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:47.736691    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:47.753722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:47.738403    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:04:48.327945 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:48.327912    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:48.328931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:48.328907    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 03 12:04:48.329009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:48.328954    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 03 12:04:49.328073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:49.328035    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 03 12:04:49.328475 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:49.328089    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 03 12:04:50.328331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:50.328294    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 03 12:04:50.328974 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:50.328941    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 03 12:04:51.328000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:51.327969    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 03 12:04:51.328175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:51.328018    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 03 12:04:52.328491 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:52.328452    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 03 12:04:52.328900 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:52.328509    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 03 12:04:52.701385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:52.701353    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 12:04:53.332625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:53.332595    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:53.333034 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:53.332915    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:53.333735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:53.333715    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:04:53.707737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:53.707686    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 12:04:55.154721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:55.136454    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:04:58.330171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.330142    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"]
Nov 03 12:04:58.334756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.334736    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:58.397234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.396661    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5951bc\""
Nov 03 12:04:58.403566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.403306    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2100fe6d-fc50-4311-abe3-0b140c4701ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:58.403566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.403390    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqs2\" (UniqueName: \"kubernetes.io/projected/2100fe6d-fc50-4311-abe3-0b140c4701ed-kube-api-access-wqqs2\") pod \"payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:58.442807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.442772    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5951bc-dockercfg-lffgb\""
Nov 03 12:04:58.444818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.444661    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"]
Nov 03 12:04:58.504888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.504847    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqs2\" (UniqueName: \"kubernetes.io/projected/2100fe6d-fc50-4311-abe3-0b140c4701ed-kube-api-access-wqqs2\") pod \"payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:58.505057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.504949    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2100fe6d-fc50-4311-abe3-0b140c4701ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:58.505563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.505542    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2100fe6d-fc50-4311-abe3-0b140c4701ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:58.557508 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.557474    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqs2\" (UniqueName: \"kubernetes.io/projected/2100fe6d-fc50-4311-abe3-0b140c4701ed-kube-api-access-wqqs2\") pod \"payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:58.666071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:58.665987    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:59.031862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:59.031836    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"]
Nov 03 12:04:59.043267 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:04:59.039002    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2100fe6d_fc50_4311_abe3_0b140c4701ed.slice/crio-cdd3778aefd3614f1ac3950a89de27eca1e074048d2b1180b33eb78481fdc79f WatchSource:0}: Error finding container cdd3778aefd3614f1ac3950a89de27eca1e074048d2b1180b33eb78481fdc79f: Status 404 returned error can't find the container with id cdd3778aefd3614f1ac3950a89de27eca1e074048d2b1180b33eb78481fdc79f
Nov 03 12:04:59.816750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:59.816717    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5" event={"ID":"2100fe6d-fc50-4311-abe3-0b140c4701ed","Type":"ContainerStarted","Data":"8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200"}
Nov 03 12:04:59.817152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:59.816775    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5" event={"ID":"2100fe6d-fc50-4311-abe3-0b140c4701ed","Type":"ContainerStarted","Data":"cdd3778aefd3614f1ac3950a89de27eca1e074048d2b1180b33eb78481fdc79f"}
Nov 03 12:04:59.817152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:59.816809    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:04:59.923202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:04:59.923138    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5" podStartSLOduration=1.923121858 podStartE2EDuration="1.923121858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:59.921521046 +0000 UTC m=+848.933987708" watchObservedRunningTime="2026-11-03 12:04:59.923121858 +0000 UTC m=+848.935588522"
Nov 03 12:05:01.868219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.868145    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"]
Nov 03 12:05:01.871017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.870998    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:01.919476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.919452    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-bb1b51-serving-cert\""
Nov 03 12:05:01.937363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.937323    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghqm\" (UniqueName: \"kubernetes.io/projected/101f028b-4d7a-4293-a325-905a16670ba4-kube-api-access-mghqm\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:01.937477 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.937375    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:01.937534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.937515    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101f028b-4d7a-4293-a325-905a16670ba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:01.937588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.937571    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-metrics-certs\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:01.967142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:01.967114    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"]
Nov 03 12:05:02.038174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.038137    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghqm\" (UniqueName: \"kubernetes.io/projected/101f028b-4d7a-4293-a325-905a16670ba4-kube-api-access-mghqm\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.038345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.038180    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.038345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.038272    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101f028b-4d7a-4293-a325-905a16670ba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.038345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.038323    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-metrics-certs\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.038345 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:05:02.038335    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-serving-cert: secret "maas-api-e2e-worker-w3-bb1b51-serving-cert" not found
Nov 03 12:05:02.038511 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:05:02.038410    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls podName:101f028b-4d7a-4293-a325-905a16670ba4 nodeName:}" failed. No retries permitted until 2026-11-03 12:05:02.538393056 +0000 UTC m=+851.550859711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls") pod "maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92" (UID: "101f028b-4d7a-4293-a325-905a16670ba4") : secret "maas-api-e2e-worker-w3-bb1b51-serving-cert" not found
Nov 03 12:05:02.038984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.038958    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101f028b-4d7a-4293-a325-905a16670ba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.040731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.040693    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-metrics-certs\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.091475 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.091450    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghqm\" (UniqueName: \"kubernetes.io/projected/101f028b-4d7a-4293-a325-905a16670ba4-kube-api-access-mghqm\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.543211 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.543160    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.545558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.545537    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:02.783561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:02.783532    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:03.147575 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.147551    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"]
Nov 03 12:05:03.148948 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:05:03.148919    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101f028b_4d7a_4293_a325_905a16670ba4.slice/crio-6d81d1c4b99622efc9614d100bc75cab605c72ffe62badf75be90bb37d91d540 WatchSource:0}: Error finding container 6d81d1c4b99622efc9614d100bc75cab605c72ffe62badf75be90bb37d91d540: Status 404 returned error can't find the container with id 6d81d1c4b99622efc9614d100bc75cab605c72ffe62badf75be90bb37d91d540
Nov 03 12:05:03.380514 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.380486    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"]
Nov 03 12:05:03.384295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.384278    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.434363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.434309    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8d2c7c-openshift-default-dockercfg-tjwjb\""
Nov 03 12:05:03.452742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.452690    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.452837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.452767    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.452837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.452827    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.452926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.452878    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.452926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.452907    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.453024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.453010    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwpx\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-kube-api-access-7rwpx\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.453063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.453040    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.453104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.453073    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.453191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.453172    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.482598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.482562    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"]
Nov 03 12:05:03.554484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554455    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.554635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554500    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.554635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554551    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.554635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554585    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.554806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554752    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.554860 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554809    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.554909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554898    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwpx\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-kube-api-access-7rwpx\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.554970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554940    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.555028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.554988    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.555289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.555261    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.555397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.555305    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.555397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.555350    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.555397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.555372    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.555526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.555442    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.556971 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.556953    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.557401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.557381    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.607838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.607812    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.608181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.608160    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwpx\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-kube-api-access-7rwpx\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.696202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.696177    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:03.837610 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.837573    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92" event={"ID":"101f028b-4d7a-4293-a325-905a16670ba4","Type":"ContainerStarted","Data":"6d81d1c4b99622efc9614d100bc75cab605c72ffe62badf75be90bb37d91d540"}
Nov 03 12:05:03.945297 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:03.945251    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:05:04.149609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.149526    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"]
Nov 03 12:05:04.169985 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:05:04.169942    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd432dc4_ee39_4b4a_a314_e37e10c834a0.slice/crio-3de84bcc90d2e951fa2a83d40d2b9ba57ccd59d1b62533270e6b50dcfbc84134 WatchSource:0}: Error finding container 3de84bcc90d2e951fa2a83d40d2b9ba57ccd59d1b62533270e6b50dcfbc84134: Status 404 returned error can't find the container with id 3de84bcc90d2e951fa2a83d40d2b9ba57ccd59d1b62533270e6b50dcfbc84134
Nov 03 12:05:04.172339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.172295    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:04.172445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.172384    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:04.172445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.172431    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:04.890757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.890692    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" event={"ID":"cd432dc4-ee39-4b4a-a314-e37e10c834a0","Type":"ContainerStarted","Data":"7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5"}
Nov 03 12:05:04.890972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.890816    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" event={"ID":"cd432dc4-ee39-4b4a-a314-e37e10c834a0","Type":"ContainerStarted","Data":"3de84bcc90d2e951fa2a83d40d2b9ba57ccd59d1b62533270e6b50dcfbc84134"}
Nov 03 12:05:04.896474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.896111    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92" event={"ID":"101f028b-4d7a-4293-a325-905a16670ba4","Type":"ContainerStarted","Data":"27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c"}
Nov 03 12:05:04.896474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:04.896168    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:05.010039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:05.009982    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" podStartSLOduration=2.009967076 podStartE2EDuration="2.009967076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:05.007534652 +0000 UTC m=+854.020001315" watchObservedRunningTime="2026-11-03 12:05:05.009967076 +0000 UTC m=+854.022433740"
Nov 03 12:05:05.122186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:05.122124    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92" podStartSLOduration=3.186935218 podStartE2EDuration="4.122107332s" totalImagesPullingTime="935.172114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:05.120125121 +0000 UTC m=+854.132591784" watchObservedRunningTime="2026-11-03 12:05:05.122107332 +0000 UTC m=+854.134573995"
Nov 03 12:05:05.696351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:05.696321    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:05.697482 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:05.697446    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 12:05:05.697534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:05.697509    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 12:05:06.697266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:06.697226    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 12:05:06.697693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:06.697279    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 12:05:07.697217 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:07.697181    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 12:05:07.697386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:07.697228    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 12:05:08.697044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:08.697008    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 12:05:08.697222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:08.697071    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 12:05:09.607934 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:05:09.607894    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:05:09.696604 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:09.696564    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 12:05:09.696791 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:09.696618    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 12:05:10.701016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:10.700983    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:10.701415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:10.701288    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:10.701877 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:10.701858    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:05:10.825633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:10.825606    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 12:05:10.908223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:10.908196    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 12:05:13.513934 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:13.513861    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:05:13.944079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:13.944046    4909 generic.go:356] "Generic (PLEG): container finished" podID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerID="c3b45b148eedde5319edd61885d7e8541b20818353e73bbadbc5242c111d136b" exitCode=0
Nov 03 12:05:13.944252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:13.944118    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" event={"ID":"12d9fb72-0575-4a4c-a47d-fd875216453c","Type":"ContainerDied","Data":"c3b45b148eedde5319edd61885d7e8541b20818353e73bbadbc5242c111d136b"}
Nov 03 12:05:13.944447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:13.944434    4909 scope.go:122] "RemoveContainer" containerID="c3b45b148eedde5319edd61885d7e8541b20818353e73bbadbc5242c111d136b"
Nov 03 12:05:13.945533 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:13.945498    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:13.945626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:13.945569    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:13.945626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:13.945600    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:14.967191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:14.967157    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" event={"ID":"12d9fb72-0575-4a4c-a47d-fd875216453c","Type":"ContainerStarted","Data":"33a5d6e8a0bca22ae18299d8afb27963d19070a8d8752280c3c7a721f38b10d3"}
Nov 03 12:05:15.614809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:15.614780    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:05:15.614988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:15.614940    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 12:05:15.615030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:15.614993    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 12:05:16.615615 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:16.615587    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 12:05:16.616081 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:16.616043    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 12:05:17.615151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:17.615116    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 12:05:17.615411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:17.615167    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 12:05:18.615901 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:18.615873    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 12:05:18.616306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:18.615924    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 12:05:19.614902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:19.614868    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 12:05:19.615088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:19.614939    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 12:05:20.625259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.625229    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:05:20.625668 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.625517    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:05:20.626116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.626099    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:05:20.661293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.661269    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"]
Nov 03 12:05:20.664674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.664658    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.712931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.712896    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bb1b51-openshift-default-dockercfg-b9g7k\""
Nov 03 12:05:20.721983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.721962    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722024    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722082    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722141    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722231    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722281    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722614    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722656    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.722758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.722686    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6np7t\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-kube-api-access-6np7t\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.763513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.763486    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"]
Nov 03 12:05:20.811043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.811017    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"]
Nov 03 12:05:20.813707 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.813681    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.823133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823113    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823143    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-home\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.823251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823160    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-dshm\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.823251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823183    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4hn\" (UniqueName: \"kubernetes.io/projected/589f3710-4603-4806-916d-d3e41a58c0f6-kube-api-access-rv4hn\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.823251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823209    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823367    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823390    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6np7t\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-kube-api-access-6np7t\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823420    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823642 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823594    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-model-cache\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.823718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823642    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589f3710-4603-4806-916d-d3e41a58c0f6-tls-certs\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.823786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823737    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823760    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823764    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823785    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-tmp-dir\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.823994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823801    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823843    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823871    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.823994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.823915    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.824217 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.824146    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.824217 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.824191    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.825421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.825403    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.825966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.825949    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.862394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.862365    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"]
Nov 03 12:05:20.862541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.862520    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"kube-root-ca.crt\""
Nov 03 12:05:20.862659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.862643    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"default-dockercfg-c46cq\""
Nov 03 12:05:20.865345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.865323    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"premium-w2-91cd99-kserve-self-signed-certs\""
Nov 03 12:05:20.880198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.880138    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6np7t\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-kube-api-access-6np7t\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.880307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.880263    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.924582 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.924554    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-home\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.924582 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.924583    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-dshm\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.924807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.924600    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4hn\" (UniqueName: \"kubernetes.io/projected/589f3710-4603-4806-916d-d3e41a58c0f6-kube-api-access-rv4hn\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.924807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.924662    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-model-cache\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.924807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.924685    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589f3710-4603-4806-916d-d3e41a58c0f6-tls-certs\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.924807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.924734    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-tmp-dir\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.925081 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.925058    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-home\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.925081 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.925077    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-tmp-dir\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.925234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.925190    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-model-cache\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.926607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.926578    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-dshm\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.926962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.926946    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589f3710-4603-4806-916d-d3e41a58c0f6-tls-certs\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:20.963964 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.962948    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"openshift-service-ca.crt\""
Nov 03 12:05:20.976795 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.976764    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:20.977794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:20.977771    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4hn\" (UniqueName: \"kubernetes.io/projected/589f3710-4603-4806-916d-d3e41a58c0f6-kube-api-access-rv4hn\") pod \"premium-w2-91cd99-kserve-85885d549c-nbvfc\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") " pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:21.134567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:21.134494    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:21.394124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:21.394096    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"]
Nov 03 12:05:21.395930 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:05:21.395902    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3af52c_4fab_4a6e_b3b4_fd75ea7b7330.slice/crio-9b792d8b80474076544ff255acce2beab2a21b5b071d900d79668b82f9399ffc WatchSource:0}: Error finding container 9b792d8b80474076544ff255acce2beab2a21b5b071d900d79668b82f9399ffc: Status 404 returned error can't find the container with id 9b792d8b80474076544ff255acce2beab2a21b5b071d900d79668b82f9399ffc
Nov 03 12:05:21.398252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:21.398222    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:21.398325 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:21.398286    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:21.398361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:21.398335    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:21.537612 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:21.537588    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"]
Nov 03 12:05:21.539852 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:05:21.539823    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod589f3710_4603_4806_916d_d3e41a58c0f6.slice/crio-7470da4113aab5c3677472101790242521dbe91ea39dbc3bac151bd4997853dc WatchSource:0}: Error finding container 7470da4113aab5c3677472101790242521dbe91ea39dbc3bac151bd4997853dc: Status 404 returned error can't find the container with id 7470da4113aab5c3677472101790242521dbe91ea39dbc3bac151bd4997853dc
Nov 03 12:05:22.004118 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.004075    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" event={"ID":"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330","Type":"ContainerStarted","Data":"01dc6c2b925d73b7244e00a3885c112de23f719bcc33317b794851ad2daf102f"}
Nov 03 12:05:22.004567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.004144    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" event={"ID":"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330","Type":"ContainerStarted","Data":"9b792d8b80474076544ff255acce2beab2a21b5b071d900d79668b82f9399ffc"}
Nov 03 12:05:22.005747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.005725    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" event={"ID":"589f3710-4603-4806-916d-d3e41a58c0f6","Type":"ContainerStarted","Data":"a84a7dccfbde364b8ae65ce90e7d86f728970a693826fa0162b35aefd16fe05d"}
Nov 03 12:05:22.005842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.005761    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" event={"ID":"589f3710-4603-4806-916d-d3e41a58c0f6","Type":"ContainerStarted","Data":"7470da4113aab5c3677472101790242521dbe91ea39dbc3bac151bd4997853dc"}
Nov 03 12:05:22.121600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.121552    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podStartSLOduration=2.121540633 podStartE2EDuration="2.121540633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:22.121347572 +0000 UTC m=+871.133814235" watchObservedRunningTime="2026-11-03 12:05:22.121540633 +0000 UTC m=+871.134007296"
Nov 03 12:05:22.134647 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.134621    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:22.228017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.227955    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podStartSLOduration=2.227936649 podStartE2EDuration="2.227936649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:22.227083808 +0000 UTC m=+871.239550470" watchObservedRunningTime="2026-11-03 12:05:22.227936649 +0000 UTC m=+871.240403314"
Nov 03 12:05:22.976982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.976946    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:22.978584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.978553    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 12:05:22.978730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:22.978606    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 12:05:23.977762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:23.977724    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 12:05:23.978187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:23.977781    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 12:05:24.977391 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:24.977353    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 12:05:24.977579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:24.977406    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 12:05:25.977563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:25.977527    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 12:05:25.978027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:25.977580    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 12:05:26.978077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:26.978039    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 12:05:26.978489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:26.978107    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 12:05:27.981804 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:27.981774    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:27.982266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:27.982141    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:27.982693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:27.982673    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:05:31.135064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:31.135030    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:31.147650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:31.147621    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:32.069768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:32.069738    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:05:32.557962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:32.557925    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:05:39.565567 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:05:39.565525    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:05:42.133226 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:42.133195    4909 generic.go:356] "Generic (PLEG): container finished" podID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerID="cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b" exitCode=0
Nov 03 12:05:42.133682 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:42.133259    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" event={"ID":"4c6815b8-75a6-4b9c-bc01-aac63d707d94","Type":"ContainerDied","Data":"cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b"}
Nov 03 12:05:42.133682 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:42.133514    4909 scope.go:122] "RemoveContainer" containerID="cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b"
Nov 03 12:05:42.134659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:42.134631    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:42.134726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:42.134709    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:42.134768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:42.134741    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:43.165068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:43.165030    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" event={"ID":"4c6815b8-75a6-4b9c-bc01-aac63d707d94","Type":"ContainerStarted","Data":"62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7"}
Nov 03 12:05:43.368377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:43.368282    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:05:43.369340 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:43.369319    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 12:05:43.369407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:43.369375    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 12:05:44.368067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:44.368029    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 12:05:44.368537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:44.368088    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 12:05:45.368517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:45.368483    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 12:05:45.368914 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:45.368536    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 12:05:46.368129 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:46.368096    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 12:05:46.368317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:46.368148    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 12:05:47.370122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:47.370082    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 12:05:47.370784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:47.370750    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 12:05:47.737566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:47.737533    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:05:48.414228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.414191    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:05:48.416657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.416632    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:05:48.669276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.669201    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"]
Nov 03 12:05:48.672578 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.672557    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.697058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.697032    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"]
Nov 03 12:05:48.700333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.700310    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.741763 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.741735    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"]
Nov 03 12:05:48.788366 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.788331    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"]
Nov 03 12:05:48.794535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794510    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk8l\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-kube-api-access-gqk8l\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.794657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794554    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3e0234-e011-4964-80e6-5e34e32ec3fe-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.794657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794621    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.794657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794646    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.794822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794670    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.794822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794694    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.794933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794832    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf01078b-da80-420e-b7ff-8a6c1febca7e-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.794933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794895    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.795007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794934    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lng2v\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-kube-api-access-lng2v\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.795007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.794982    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.795077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795017    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.795077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795056    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.795153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795131    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.795201 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795151    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.795201 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795192    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.795291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795225    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.795291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795256    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.795291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.795282    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.896364 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896333    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.896497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896394    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.896497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896427    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.896497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896451    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.896497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896485    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.896693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896512    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.896693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896539    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.896693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896574    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.896693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896606    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk8l\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-kube-api-access-gqk8l\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.896693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896653    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3e0234-e011-4964-80e6-5e34e32ec3fe-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.896693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896692    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896772    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896806    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896827    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896843    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896925    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf01078b-da80-420e-b7ff-8a6c1febca7e-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896966    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.896995    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lng2v\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-kube-api-access-lng2v\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.897012    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.897029    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.897039    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897548 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.897057    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.897722    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.897902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.897819    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.898002 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.897976    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.898122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.898036    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf01078b-da80-420e-b7ff-8a6c1febca7e-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.898122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.898083    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.898371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.898352    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3e0234-e011-4964-80e6-5e34e32ec3fe-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.899134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.899112    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.899474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.899452    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.899557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.899510    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.899668 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.899653    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.949742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.949712    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.949932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.949914    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lng2v\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-kube-api-access-lng2v\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:48.949993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.949927    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.950202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.950181    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk8l\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-kube-api-access-gqk8l\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:48.989154 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:48.989132    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:49.012731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.012708    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:49.412876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.412849    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"]
Nov 03 12:05:49.412977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.412907    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"]
Nov 03 12:05:49.414618 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:05:49.414591    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3e0234_e011_4964_80e6_5e34e32ec3fe.slice/crio-99590baa8dc63cbaae895561cb8d74a989ed82ef76f59687a789cc6af2d4d106 WatchSource:0}: Error finding container 99590baa8dc63cbaae895561cb8d74a989ed82ef76f59687a789cc6af2d4d106: Status 404 returned error can't find the container with id 99590baa8dc63cbaae895561cb8d74a989ed82ef76f59687a789cc6af2d4d106
Nov 03 12:05:49.415314 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:05:49.415294    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf01078b_da80_420e_b7ff_8a6c1febca7e.slice/crio-20947c8054ecc9bfc6b580cc660101e9d8f2d6003a109df023847e1c427d28a5 WatchSource:0}: Error finding container 20947c8054ecc9bfc6b580cc660101e9d8f2d6003a109df023847e1c427d28a5: Status 404 returned error can't find the container with id 20947c8054ecc9bfc6b580cc660101e9d8f2d6003a109df023847e1c427d28a5
Nov 03 12:05:49.416987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.416956    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:49.417070 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.417023    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:49.417070 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.417066    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:49.417828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.417787    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:49.417920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.417860    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:49.417920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:49.417904    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:05:50.201506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.201463    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" event={"ID":"bf01078b-da80-420e-b7ff-8a6c1febca7e","Type":"ContainerStarted","Data":"1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125"}
Nov 03 12:05:50.201728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.201527    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" event={"ID":"bf01078b-da80-420e-b7ff-8a6c1febca7e","Type":"ContainerStarted","Data":"20947c8054ecc9bfc6b580cc660101e9d8f2d6003a109df023847e1c427d28a5"}
Nov 03 12:05:50.203163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.203132    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" event={"ID":"de3e0234-e011-4964-80e6-5e34e32ec3fe","Type":"ContainerStarted","Data":"b958e01c9e94edea55c691b336aa73e1ae77b79d4d2cbae9631761313327637e"}
Nov 03 12:05:50.203287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.203179    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" event={"ID":"de3e0234-e011-4964-80e6-5e34e32ec3fe","Type":"ContainerStarted","Data":"99590baa8dc63cbaae895561cb8d74a989ed82ef76f59687a789cc6af2d4d106"}
Nov 03 12:05:50.414295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.414243    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podStartSLOduration=2.414226685 podStartE2EDuration="2.414226685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:50.368202181 +0000 UTC m=+899.380668845" watchObservedRunningTime="2026-11-03 12:05:50.414226685 +0000 UTC m=+899.426693347"
Nov 03 12:05:50.990066 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.990018    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:50.991456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.991433    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 12:05:50.991589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:50.991492    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 12:05:51.013670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:51.013639    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:51.014685 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:51.014662    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 12:05:51.014831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:51.014718    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 12:05:51.216846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:51.216818    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:05:51.219004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:51.218978    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:05:51.990107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:51.990068    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 12:05:51.990647 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:51.990133    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 12:05:52.013491 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:52.013463    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 12:05:52.013633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:52.013511    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 12:05:52.991042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:52.991007    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 12:05:52.991638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:52.991609    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 12:05:53.013536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:53.013514    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 12:05:53.013687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:53.013671    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 12:05:53.990257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:53.990207    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 12:05:53.990416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:53.990265    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 12:05:54.014336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:54.014107    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 12:05:54.014336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:54.014166    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 12:05:54.990006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:54.989961    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 12:05:54.990222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:54.990022    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 12:05:55.013998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:55.013966    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 12:05:55.014147 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:55.014020    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 12:05:55.993905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:55.993875    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:55.994304 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:55.994121    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:55.994788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:55.994771    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:05:56.016941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:56.016922    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:56.017197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:56.017180    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:56.017918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:56.017900    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 12:05:56.148963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:56.148911    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podStartSLOduration=8.148894652 podStartE2EDuration="8.148894652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:50.487916691 +0000 UTC m=+899.500383359" watchObservedRunningTime="2026-11-03 12:05:56.148894652 +0000 UTC m=+905.161361316"
Nov 03 12:05:58.923834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:05:58.923799    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:06:05.219741 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.219713    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"]
Nov 03 12:06:05.223200 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.223182    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.318619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.318588    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-dshm\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.318786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.318635    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ds9l\" (UniqueName: \"kubernetes.io/projected/d8e314af-4ea2-419e-879c-eaa78cedb602-kube-api-access-8ds9l\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.318786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.318772    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e314af-4ea2-419e-879c-eaa78cedb602-tls-certs\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.318905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.318882    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-model-cache\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.318940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.318922    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-home\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.318971 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.318954    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-tmp-dir\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.320589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.320566    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"kube-root-ca.crt\""
Nov 03 12:06:05.320790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.320773    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"openshift-service-ca.crt\""
Nov 03 12:06:05.320842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.320820    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"default-dockercfg-87hfd\""
Nov 03 12:06:05.320888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.320849    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"simulator-w3-bb1b51-kserve-self-signed-certs\""
Nov 03 12:06:05.322727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.322707    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"]
Nov 03 12:06:05.419785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.419746    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e314af-4ea2-419e-879c-eaa78cedb602-tls-certs\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.419938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.419834    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-model-cache\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.419938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.419856    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-home\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.419938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.419872    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-tmp-dir\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.419938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.419897    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-dshm\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.419938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.419923    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ds9l\" (UniqueName: \"kubernetes.io/projected/d8e314af-4ea2-419e-879c-eaa78cedb602-kube-api-access-8ds9l\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.420342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.420306    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-home\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.420342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.420332    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-model-cache\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.420500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.420370    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-tmp-dir\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.422114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.422096    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-dshm\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.422488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.422472    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e314af-4ea2-419e-879c-eaa78cedb602-tls-certs\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.476458 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.476400    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ds9l\" (UniqueName: \"kubernetes.io/projected/d8e314af-4ea2-419e-879c-eaa78cedb602-kube-api-access-8ds9l\") pod \"simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") " pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.535038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.535012    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:05.896941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:05.896920    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"]
Nov 03 12:06:05.898881 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:06:05.898851    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8e314af_4ea2_419e_879c_eaa78cedb602.slice/crio-84e1c4c17643545ff1c7df293bcdefca61aac3d715a5477d4927636e77e85cc6 WatchSource:0}: Error finding container 84e1c4c17643545ff1c7df293bcdefca61aac3d715a5477d4927636e77e85cc6: Status 404 returned error can't find the container with id 84e1c4c17643545ff1c7df293bcdefca61aac3d715a5477d4927636e77e85cc6
Nov 03 12:06:06.006560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.006531    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"]
Nov 03 12:06:06.009744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.009725    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.066308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.066282    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"]
Nov 03 12:06:06.125568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125541    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df49b-98a6-46dd-9f16-64466785b187-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.125736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125585    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4klb\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-kube-api-access-b4klb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.125736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125609    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.125851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125738    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.125851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125761    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.125851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125811    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.125973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125902    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.126009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125976    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.126009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.125995    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df49b-98a6-46dd-9f16-64466785b187-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.226870 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.226843    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.226884    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.226924    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.226940    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df49b-98a6-46dd-9f16-64466785b187-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.226976    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df49b-98a6-46dd-9f16-64466785b187-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227054    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4klb\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-kube-api-access-b4klb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227099    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227149    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227178    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227467    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227502    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227635    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227727    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df49b-98a6-46dd-9f16-64466785b187-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.227878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.227784    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.229213 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.229191    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.229429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.229413    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df49b-98a6-46dd-9f16-64466785b187-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.280078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.280055    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4klb\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-kube-api-access-b4klb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.280753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.280737    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.289634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.289612    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" event={"ID":"d8e314af-4ea2-419e-879c-eaa78cedb602","Type":"ContainerStarted","Data":"7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822"}
Nov 03 12:06:06.289761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.289646    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" event={"ID":"d8e314af-4ea2-419e-879c-eaa78cedb602","Type":"ContainerStarted","Data":"84e1c4c17643545ff1c7df293bcdefca61aac3d715a5477d4927636e77e85cc6"}
Nov 03 12:06:06.321807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.321788    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:06.397378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.397328    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podStartSLOduration=1.397312363 podStartE2EDuration="1.397312363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:06.395007855 +0000 UTC m=+915.407474530" watchObservedRunningTime="2026-11-03 12:06:06.397312363 +0000 UTC m=+915.409779027"
Nov 03 12:06:06.545642 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.535922    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:06.734738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.734713    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"]
Nov 03 12:06:06.736826 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:06:06.736789    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62df49b_98a6_46dd_9f16_64466785b187.slice/crio-0170c7ee8ba86b4492894260f228bd97cf323c7f2a92eab0459533e848b8c42e WatchSource:0}: Error finding container 0170c7ee8ba86b4492894260f228bd97cf323c7f2a92eab0459533e848b8c42e: Status 404 returned error can't find the container with id 0170c7ee8ba86b4492894260f228bd97cf323c7f2a92eab0459533e848b8c42e
Nov 03 12:06:06.740543 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.740511    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:06:06.740633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.740573    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:06:06.740633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:06.740599    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:06:07.295971 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:07.295930    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" event={"ID":"f62df49b-98a6-46dd-9f16-64466785b187","Type":"ContainerStarted","Data":"42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604"}
Nov 03 12:06:07.296368 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:07.295988    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" event={"ID":"f62df49b-98a6-46dd-9f16-64466785b187","Type":"ContainerStarted","Data":"0170c7ee8ba86b4492894260f228bd97cf323c7f2a92eab0459533e848b8c42e"}
Nov 03 12:06:07.322425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:07.322396    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:07.324194 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:07.324164    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:06:07.324302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:07.324211    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 12:06:07.498641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:07.498584    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podStartSLOduration=2.498570207 podStartE2EDuration="2.498570207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:07.463166323 +0000 UTC m=+916.475632998" watchObservedRunningTime="2026-11-03 12:06:07.498570207 +0000 UTC m=+916.511036869"
Nov 03 12:06:08.322941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:08.322907    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:06:08.323350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:08.322951    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 12:06:09.322600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:09.322567    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:06:09.322792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:09.322616    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 12:06:09.589120 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:06:09.589041    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:06:10.322647 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:10.322612    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:06:10.322856 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:10.322670    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 12:06:11.322828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:11.322794    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:06:11.323217 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:11.322853    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 12:06:12.323082 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:12.323047    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:06:12.323496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:12.323098    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 12:06:13.326556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:13.326526    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:13.326976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:13.326791    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:13.327439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:13.327419    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 12:06:15.535441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:15.535406    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:15.548352 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:15.548327    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:16.360267 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:16.360236    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:06:20.103805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:20.103770    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:06:37.690669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:06:37.690578    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:06:39.565917 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:06:39.565882    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:07:03.015828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:03.015797    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:07:09.564278 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:07:09.564241    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:07:22.575546 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.575418    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"]
Nov 03 12:07:22.580280 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.580254    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.628162 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.628129    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl7w4\" (UniqueName: \"kubernetes.io/projected/c7f1faf3-c922-46e8-b9a9-c68465c47a93-kube-api-access-hl7w4\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.628337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.628173    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-model-cache\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.628414 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.628333    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tls-certs\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.628414 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.628371    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-dshm\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.628414 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.628395    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tmp-dir\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.628574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.628514    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-home\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.674976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.674950    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"openshift-service-ca.crt\""
Nov 03 12:07:22.675121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.675025    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"default-dockercfg-lk5jw\""
Nov 03 12:07:22.675258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.675242    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"premium-w1-5951bc-kserve-self-signed-certs\""
Nov 03 12:07:22.675258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.675247    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"kube-root-ca.crt\""
Nov 03 12:07:22.677292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.677273    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"]
Nov 03 12:07:22.730911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.730875    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl7w4\" (UniqueName: \"kubernetes.io/projected/c7f1faf3-c922-46e8-b9a9-c68465c47a93-kube-api-access-hl7w4\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.731161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.731140    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-model-cache\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.731358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.731338    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tls-certs\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.731478 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.731464    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-dshm\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.731577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.731562    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tmp-dir\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.731761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.731694    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-home\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.732594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.732572    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-home\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.736056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.735235    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-dshm\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.736056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.735491    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tmp-dir\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.736056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.736017    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-model-cache\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.743865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.743839    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tls-certs\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.788683 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.788655    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl7w4\" (UniqueName: \"kubernetes.io/projected/c7f1faf3-c922-46e8-b9a9-c68465c47a93-kube-api-access-hl7w4\") pod \"premium-w1-5951bc-kserve-84c4dddb68-mfwt8\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") " pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:22.891849 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:22.891783    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:23.297236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:23.297213    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"]
Nov 03 12:07:23.299377 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:07:23.299349    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7f1faf3_c922_46e8_b9a9_c68465c47a93.slice/crio-d9ebf99ede9537fbb748ddbae05b6f2031ba78132bf54568c1e7c05a7c8b7ee7 WatchSource:0}: Error finding container d9ebf99ede9537fbb748ddbae05b6f2031ba78132bf54568c1e7c05a7c8b7ee7: Status 404 returned error can't find the container with id d9ebf99ede9537fbb748ddbae05b6f2031ba78132bf54568c1e7c05a7c8b7ee7
Nov 03 12:07:23.783990 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:23.783955    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" event={"ID":"c7f1faf3-c922-46e8-b9a9-c68465c47a93","Type":"ContainerStarted","Data":"bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e"}
Nov 03 12:07:23.784350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:23.784003    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" event={"ID":"c7f1faf3-c922-46e8-b9a9-c68465c47a93","Type":"ContainerStarted","Data":"d9ebf99ede9537fbb748ddbae05b6f2031ba78132bf54568c1e7c05a7c8b7ee7"}
Nov 03 12:07:23.890268 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:23.890221    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podStartSLOduration=1.890210289 podStartE2EDuration="1.890210289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:07:23.889993335 +0000 UTC m=+992.902459998" watchObservedRunningTime="2026-11-03 12:07:23.890210289 +0000 UTC m=+992.902676951"
Nov 03 12:07:23.892048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:23.892029    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:32.739361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:32.739328    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:07:32.892523 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:32.892495    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:32.905257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:32.905231    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:33.865921 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:07:33.865887    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:07:39.564307 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:07:39.564266    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:08:09.563136 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:08:09.563056    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:08:12.485143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:12.485105    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:08:29.197326 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.197294    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"]
Nov 03 12:08:29.201174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.201153    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.263771 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.263732    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69x74\" (UniqueName: \"kubernetes.io/projected/6852b48a-9d35-4116-8d6f-da6ecfb018ea-kube-api-access-69x74\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.263929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.263795    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-home\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.263929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.263846    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-dshm\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.264008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.263931    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tmp-dir\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.264008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.263995    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tls-certs\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.264082 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.264051    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-model-cache\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.298257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.298228    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"e2e-conflict-a-78b30cda-kserve-self-signed-certs\""
Nov 03 12:08:29.298257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.298230    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"default-dockercfg-n4q48\""
Nov 03 12:08:29.298459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.298267    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"openshift-service-ca.crt\""
Nov 03 12:08:29.298459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.298274    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"kube-root-ca.crt\""
Nov 03 12:08:29.300680 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.300658    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"]
Nov 03 12:08:29.375263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.370325    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69x74\" (UniqueName: \"kubernetes.io/projected/6852b48a-9d35-4116-8d6f-da6ecfb018ea-kube-api-access-69x74\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.375263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.370401    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-home\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.375263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.370444    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-dshm\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.375263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.370514    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tmp-dir\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.375263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.370595    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tls-certs\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.375263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.370643    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-model-cache\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.376298 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.376261    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-model-cache\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.377042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.377019    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-home\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.378341 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.378317    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tmp-dir\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.388719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.388624    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-dshm\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.391860 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.390343    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tls-certs\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.428831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.428807    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69x74\" (UniqueName: \"kubernetes.io/projected/6852b48a-9d35-4116-8d6f-da6ecfb018ea-kube-api-access-69x74\") pod \"e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.512867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.512795    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:29.929912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:29.929883    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"]
Nov 03 12:08:29.931684 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:08:29.931656    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6852b48a_9d35_4116_8d6f_da6ecfb018ea.slice/crio-b1c7828be955d33bcd59fbb08db4f1baeeec6950fde009773420e49ee385ae46 WatchSource:0}: Error finding container b1c7828be955d33bcd59fbb08db4f1baeeec6950fde009773420e49ee385ae46: Status 404 returned error can't find the container with id b1c7828be955d33bcd59fbb08db4f1baeeec6950fde009773420e49ee385ae46
Nov 03 12:08:30.245586 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:30.245551    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v" event={"ID":"6852b48a-9d35-4116-8d6f-da6ecfb018ea","Type":"ContainerStarted","Data":"b538b78baf080032e2350bf277afdbfc9d5b4e9713a161ce18dd7c4713dcb5fc"}
Nov 03 12:08:30.245962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:30.245592    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v" event={"ID":"6852b48a-9d35-4116-8d6f-da6ecfb018ea","Type":"ContainerStarted","Data":"b1c7828be955d33bcd59fbb08db4f1baeeec6950fde009773420e49ee385ae46"}
Nov 03 12:08:30.352028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:30.351966    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v" podStartSLOduration=1.35195258 podStartE2EDuration="1.35195258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:30.350848448 +0000 UTC m=+1059.363315112" watchObservedRunningTime="2026-11-03 12:08:30.35195258 +0000 UTC m=+1059.364419243"
Nov 03 12:08:30.513585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:30.513503    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:35.704905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.704874    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"]
Nov 03 12:08:35.708673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.708654    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.805581 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.805549    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"]
Nov 03 12:08:35.854744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.854693    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.854882 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.854758    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbpj\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-kube-api-access-bpbpj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.854882 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.854846    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.854974 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.854930    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.855032 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.854974    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.855124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.855107    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.855185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.855157    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.855234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.855213    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.855380 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.855358    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956048    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956088    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956104    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956192    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956218    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956251    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbpj\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-kube-api-access-bpbpj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956297    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956393    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956416    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956501    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.956955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.956931    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.957259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.957234    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.957342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.957308    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.957424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.957406    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.958841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.958821    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:35.959053 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:35.959036    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:36.008651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:36.008628    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:36.008752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:36.008662    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbpj\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-kube-api-access-bpbpj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzpnv\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:36.021659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:36.021634    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:36.433541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:36.433513    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"]
Nov 03 12:08:36.434998 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:08:36.434970    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40b512a_d8d7_4113_a10c_7b0fb47bbcbc.slice/crio-387f89beaf8ca306e2f95271bba8d0582e15cc9ed3547342e6499a6ba4860ae9 WatchSource:0}: Error finding container 387f89beaf8ca306e2f95271bba8d0582e15cc9ed3547342e6499a6ba4860ae9: Status 404 returned error can't find the container with id 387f89beaf8ca306e2f95271bba8d0582e15cc9ed3547342e6499a6ba4860ae9
Nov 03 12:08:36.437260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:36.437226    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:08:36.437345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:36.437298    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:08:36.437345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:36.437328    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:08:37.284138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:37.284098    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" event={"ID":"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc","Type":"ContainerStarted","Data":"4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41"}
Nov 03 12:08:37.284530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:37.284163    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" event={"ID":"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc","Type":"ContainerStarted","Data":"387f89beaf8ca306e2f95271bba8d0582e15cc9ed3547342e6499a6ba4860ae9"}
Nov 03 12:08:37.441724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:37.438350    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podStartSLOduration=2.438332319 podStartE2EDuration="2.438332319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:37.436305126 +0000 UTC m=+1066.448771789" watchObservedRunningTime="2026-11-03 12:08:37.438332319 +0000 UTC m=+1066.450798982"
Nov 03 12:08:38.022732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:38.022679    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:38.024434 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:38.024407    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:08:38.024572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:38.024461    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 12:08:39.022464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:39.022429    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:08:39.022899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:39.022483    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 12:08:39.513470 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:39.513434    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:39.526620 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:39.526590    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:39.588013 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:08:39.587977    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:08:40.022752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:40.022716    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:08:40.023235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:40.022779    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 12:08:40.311153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:40.311076    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:08:41.022141 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:41.022110    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:08:41.022292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:41.022166    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 12:08:42.022079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:42.022044    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:08:42.022456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:42.022099    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 12:08:43.022415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:43.022381    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:08:43.022829 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:43.022429    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 12:08:43.554981 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:43.554952    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:08:44.022611 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:44.022567    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:08:44.023123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:44.022631    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 12:08:46.023363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:46.023317    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:08:46.023902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:46.023376    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:08:47.024159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:47.024112    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:08:47.024554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:47.024173    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:08:47.025236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:47.025215    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:47.025631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:47.025614    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:47.026411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:47.026394    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:08:47.168914 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:47.168859    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:08:47.169279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:47.169218    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:08:50.819421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.819388    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"]
Nov 03 12:08:50.823804 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.823783    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890504    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890549    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9952d74a-882b-476d-a0cb-53d663693c1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890628    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890682    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfw7\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-kube-api-access-zpfw7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890732    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890786    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890895 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890863    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9952d74a-882b-476d-a0cb-53d663693c1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890900    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.890982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.890952    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.915172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.915145    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"]
Nov 03 12:08:50.924565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.924539    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"]
Nov 03 12:08:50.931845 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.931823    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:50.980749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.980694    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4e5279dc91ae\""
Nov 03 12:08:50.980749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.980738    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:08:50.980930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.980752    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:08:50.980930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.980748    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:08:50.980930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.980869    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-x4r42\""
Nov 03 12:08:50.980930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.980908    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:08:50.992572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992547    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.992645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992583    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-routing-overlay-volume\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:50.992645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992601    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-provider-credentials\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:50.992645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992628    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9952d74a-882b-476d-a0cb-53d663693c1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.992818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992793    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.992875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992849    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-plugins-config-volume\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:50.992926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992883    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9j4\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-kube-api-access-fv9j4\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:50.992926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992901    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.992923    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfw7\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-kube-api-access-zpfw7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993059 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993027    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993079    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993139 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993105    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9952d74a-882b-476d-a0cb-53d663693c1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993136    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993185    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993398    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993462 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993422    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993543    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.993814 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.993795    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9952d74a-882b-476d-a0cb-53d663693c1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.995050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.995028    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9952d74a-882b-476d-a0cb-53d663693c1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:50.995323 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:50.995304    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:51.011289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.011268    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"]
Nov 03 12:08:51.045307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.045277    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:51.045501 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.045483    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfw7\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-kube-api-access-zpfw7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnmsr\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:51.094231 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.094172    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-routing-overlay-volume\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.094231 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.094202    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-provider-credentials\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.094403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.094242    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-plugins-config-volume\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.094403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.094271    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9j4\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-kube-api-access-fv9j4\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.136662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.136634    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:51.141767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.141745    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:08:51.141767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.141753    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4e5279dc91ae\""
Nov 03 12:08:51.141932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.141769    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:08:51.145431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.145407    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-routing-overlay-volume\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.145544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.145438    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-plugins-config-volume\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.147055 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.147034    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-provider-credentials\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.197264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.197240    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:08:51.244430 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.244396    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:08:51.249780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.249756    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9j4\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-kube-api-access-fv9j4\") pod \"payload-processing-external-model-59f7bdbbd8-hlgbr\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") " pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.550752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.550729    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"]
Nov 03 12:08:51.552847 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:08:51.552816    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9952d74a_882b_476d_a0cb_53d663693c1e.slice/crio-ca88eb69f8294401d4fa5e6f8766f69901a23741bb90dbd3132728a64d1ef8cf WatchSource:0}: Error finding container ca88eb69f8294401d4fa5e6f8766f69901a23741bb90dbd3132728a64d1ef8cf: Status 404 returned error can't find the container with id ca88eb69f8294401d4fa5e6f8766f69901a23741bb90dbd3132728a64d1ef8cf
Nov 03 12:08:51.555068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.555038    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:08:51.555134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.555100    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:08:51.555134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.555128    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:08:51.592557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.592536    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-x4r42\""
Nov 03 12:08:51.595389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.595362    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:51.967223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:51.967195    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"]
Nov 03 12:08:51.969349 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:08:51.969319    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0893a389_fe00_4ab9_89d5_245764057208.slice/crio-bf1560815a2ffa956e7cc68c2ade101f6e51c66585b2c73137f48861e98bd754 WatchSource:0}: Error finding container bf1560815a2ffa956e7cc68c2ade101f6e51c66585b2c73137f48861e98bd754: Status 404 returned error can't find the container with id bf1560815a2ffa956e7cc68c2ade101f6e51c66585b2c73137f48861e98bd754
Nov 03 12:08:52.495764 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:52.495721    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" event={"ID":"0893a389-fe00-4ab9-89d5-245764057208","Type":"ContainerStarted","Data":"cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295"}
Nov 03 12:08:52.495958 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:52.495825    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" event={"ID":"0893a389-fe00-4ab9-89d5-245764057208","Type":"ContainerStarted","Data":"bf1560815a2ffa956e7cc68c2ade101f6e51c66585b2c73137f48861e98bd754"}
Nov 03 12:08:52.495958 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:52.495861    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:08:52.497199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:52.497175    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" event={"ID":"9952d74a-882b-476d-a0cb-53d663693c1e","Type":"ContainerStarted","Data":"75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc"}
Nov 03 12:08:52.497365 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:52.497213    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" event={"ID":"9952d74a-882b-476d-a0cb-53d663693c1e","Type":"ContainerStarted","Data":"ca88eb69f8294401d4fa5e6f8766f69901a23741bb90dbd3132728a64d1ef8cf"}
Nov 03 12:08:52.615530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:52.615479    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" podStartSLOduration=2.61546636 podStartE2EDuration="2.61546636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:52.613988831 +0000 UTC m=+1081.626455494" watchObservedRunningTime="2026-11-03 12:08:52.61546636 +0000 UTC m=+1081.627933023"
Nov 03 12:08:52.724358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:52.724308    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podStartSLOduration=2.724293741 podStartE2EDuration="2.724293741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:52.722658596 +0000 UTC m=+1081.735125261" watchObservedRunningTime="2026-11-03 12:08:52.724293741 +0000 UTC m=+1081.736760403"
Nov 03 12:08:53.137267 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:53.137229    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:08:53.160722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:53.156982    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 03 12:08:53.160722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:53.157043    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 03 12:08:54.137630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:54.137600    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 03 12:08:54.138128 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:54.137648    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 03 12:08:55.137479 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:55.137449    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 03 12:08:55.137660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:55.137498    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 03 12:08:56.137383 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:56.137348    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 03 12:08:56.137532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:56.137407    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 03 12:08:57.137382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:57.137346    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 03 12:08:57.137811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:57.137404    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 03 12:08:58.137366 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:58.137330    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 03 12:08:58.137565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:58.137386    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 03 12:08:59.137498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:59.137467    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 03 12:08:59.137670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:08:59.137515    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 03 12:09:00.141407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:00.141377    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:09:00.141901 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:00.141735    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:09:00.142521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:00.142502    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:09:03.074731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:03.073861    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:09:03.074731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:03.073940    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:03.150991 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:03.150832    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:09:03.150991 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:03.150915    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:03.535987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:03.535796    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:09:05.755080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.755048    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"]
Nov 03 12:09:05.759153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.759133    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.855393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.855361    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"distinct-w1-5951bc-kserve-self-signed-certs\""
Nov 03 12:09:05.857977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.857944    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"]
Nov 03 12:09:05.858447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.858420    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-model-cache\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.858534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.858500    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-home\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.858595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.858550    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tls-certs\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.858692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.858671    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-dshm\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.858752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.858719    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tmp-dir\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.858790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.858775    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6j2\" (UniqueName: \"kubernetes.io/projected/4120763b-a5bc-481d-a9ef-7d178c40b7b8-kube-api-access-2f6j2\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.964721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.964452    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-model-cache\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.964721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.964554    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-home\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.964721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.964617    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tls-certs\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.964721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.964684    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-dshm\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.965080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.964735    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tmp-dir\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.965080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.964766    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6j2\" (UniqueName: \"kubernetes.io/projected/4120763b-a5bc-481d-a9ef-7d178c40b7b8-kube-api-access-2f6j2\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.965080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.964934    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-model-cache\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.965244 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.965226    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-home\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.965551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.965490    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tmp-dir\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.969115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.969092    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-dshm\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:05.969897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:05.969875    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tls-certs\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:06.017755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:06.017687    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6j2\" (UniqueName: \"kubernetes.io/projected/4120763b-a5bc-481d-a9ef-7d178c40b7b8-kube-api-access-2f6j2\") pod \"distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:06.071261 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:06.071229    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:06.470074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:06.470044    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"]
Nov 03 12:09:06.471592 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:09:06.471561    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4120763b_a5bc_481d_a9ef_7d178c40b7b8.slice/crio-80ec2e9cb774983ffd0fdbf93de54897ac137335a1eabe498a2d95ab0d8a9ba1 WatchSource:0}: Error finding container 80ec2e9cb774983ffd0fdbf93de54897ac137335a1eabe498a2d95ab0d8a9ba1: Status 404 returned error can't find the container with id 80ec2e9cb774983ffd0fdbf93de54897ac137335a1eabe498a2d95ab0d8a9ba1
Nov 03 12:09:06.652530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:06.652496    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" event={"ID":"4120763b-a5bc-481d-a9ef-7d178c40b7b8","Type":"ContainerStarted","Data":"3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359"}
Nov 03 12:09:06.652738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:06.652541    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" event={"ID":"4120763b-a5bc-481d-a9ef-7d178c40b7b8","Type":"ContainerStarted","Data":"80ec2e9cb774983ffd0fdbf93de54897ac137335a1eabe498a2d95ab0d8a9ba1"}
Nov 03 12:09:06.758864 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:06.758767    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podStartSLOduration=1.7587540069999998 podStartE2EDuration="1.758754007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:06.756974365 +0000 UTC m=+1095.769441028" watchObservedRunningTime="2026-11-03 12:09:06.758754007 +0000 UTC m=+1095.771220670"
Nov 03 12:09:07.071797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:07.071678    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:09.563998 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:09:09.563966    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:09:12.332049 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:12.332018    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:09:13.505545 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:13.505509    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:09:16.072254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:16.072218    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:16.084471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:16.084446    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:16.737529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:16.737500    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:09:23.505291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:23.505257    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:09:28.374193 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:28.374142    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"]
Nov 03 12:09:28.375877 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:28.375844    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:09:33.375388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.375300    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing" containerID="cri-o://cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295" gracePeriod=25
Nov 03 12:09:33.630522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.630466    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:09:33.766551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.766518    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-plugins-config-volume\") pod \"0893a389-fe00-4ab9-89d5-245764057208\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") "
Nov 03 12:09:33.766747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.766587    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-kube-api-access-fv9j4\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-kube-api-access-fv9j4\") pod \"0893a389-fe00-4ab9-89d5-245764057208\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") "
Nov 03 12:09:33.766747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.766636    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-routing-overlay-volume\") pod \"0893a389-fe00-4ab9-89d5-245764057208\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") "
Nov 03 12:09:33.766897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.766806    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-provider-credentials\") pod \"0893a389-fe00-4ab9-89d5-245764057208\" (UID: \"0893a389-fe00-4ab9-89d5-245764057208\") "
Nov 03 12:09:33.767039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.766998    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-plugins-config-volume" pod "0893a389-fe00-4ab9-89d5-245764057208" (UID: "0893a389-fe00-4ab9-89d5-245764057208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:09:33.767140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.767045    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-routing-overlay-volume" pod "0893a389-fe00-4ab9-89d5-245764057208" (UID: "0893a389-fe00-4ab9-89d5-245764057208"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:09:33.767300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.767281    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:33.767404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.767345    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0893a389-fe00-4ab9-89d5-245764057208-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:33.769523 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.769498    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-provider-credentials" pod "0893a389-fe00-4ab9-89d5-245764057208" (UID: "0893a389-fe00-4ab9-89d5-245764057208"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:09:33.769635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.769555    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-kube-api-access-fv9j4" pod "0893a389-fe00-4ab9-89d5-245764057208" (UID: "0893a389-fe00-4ab9-89d5-245764057208"). InnerVolumeSpecName "kube-api-access-fv9j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:09:33.854179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.854147    4909 generic.go:356] "Generic (PLEG): container finished" podID="0893a389-fe00-4ab9-89d5-245764057208" containerID="cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295" exitCode=0
Nov 03 12:09:33.854353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.854200    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"
Nov 03 12:09:33.854353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.854231    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" event={"ID":"0893a389-fe00-4ab9-89d5-245764057208","Type":"ContainerDied","Data":"cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295"}
Nov 03 12:09:33.854353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.854286    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr" event={"ID":"0893a389-fe00-4ab9-89d5-245764057208","Type":"ContainerDied","Data":"bf1560815a2ffa956e7cc68c2ade101f6e51c66585b2c73137f48861e98bd754"}
Nov 03 12:09:33.854353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.854311    4909 scope.go:122] "RemoveContainer" containerID="cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295"
Nov 03 12:09:33.866304 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.866223    4909 scope.go:122] "RemoveContainer" containerID="cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295"
Nov 03 12:09:33.866767 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:09:33.866738    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295\": container with ID starting with cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295 not found: ID does not exist" containerID="cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295"
Nov 03 12:09:33.866866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.866778    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295"} err="failed to get container status \"cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295\": rpc error: code = NotFound desc = could not find container \"cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295\": container with ID starting with cd0c25820560ab94037b9bb4734e4b42e44d8e43ff503dade762ed122600e295 not found: ID does not exist"
Nov 03 12:09:33.869001 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.868977    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9j4\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-kube-api-access-fv9j4\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:33.869108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:33.869005    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0893a389-fe00-4ab9-89d5-245764057208-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:34.005709 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:34.005667    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"]
Nov 03 12:09:34.011207 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:34.011184    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f7bdbbd8-hlgbr"]
Nov 03 12:09:35.123522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:35.123482    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0893a389-fe00-4ab9-89d5-245764057208" path="/var/lib/kubelet/pods/0893a389-fe00-4ab9-89d5-245764057208/volumes"
Nov 03 12:09:36.983398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:36.983365    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"]
Nov 03 12:09:36.984073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:36.984046    4909 cpu_manager.go:414] "containerMap: removing container" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing"
Nov 03 12:09:36.984073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:36.984065    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing"
Nov 03 12:09:36.984228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:36.984179    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="0893a389-fe00-4ab9-89d5-245764057208" containerName="payload-processing"
Nov 03 12:09:36.987349 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:36.987332    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.083472 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.083433    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"e2e-conflict-b-78b30cda-kserve-self-signed-certs\""
Nov 03 12:09:37.085979 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.085955    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"]
Nov 03 12:09:37.103994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.103968    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-home\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.104140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.104118    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-tmp-dir\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.104211 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.104156    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7rl\" (UniqueName: \"kubernetes.io/projected/b1973fb8-8a5f-458a-8695-b68d91bfea39-kube-api-access-fq7rl\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.104315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.104302    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-dshm\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.104379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.104329    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-model-cache\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.104429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.104412    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1973fb8-8a5f-458a-8695-b68d91bfea39-tls-certs\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.208480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.208440    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-tmp-dir\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.208480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.208480    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7rl\" (UniqueName: \"kubernetes.io/projected/b1973fb8-8a5f-458a-8695-b68d91bfea39-kube-api-access-fq7rl\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.208760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.208562    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-dshm\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.208760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.208579    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-model-cache\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.208760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.208617    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1973fb8-8a5f-458a-8695-b68d91bfea39-tls-certs\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.208760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.208688    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-home\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.209421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.209401    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-home\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.209569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.209458    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-model-cache\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.210174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.210155    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-tmp-dir\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.212468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.212443    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-dshm\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.213371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.213350    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1973fb8-8a5f-458a-8695-b68d91bfea39-tls-certs\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.261638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.261582    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7rl\" (UniqueName: \"kubernetes.io/projected/b1973fb8-8a5f-458a-8695-b68d91bfea39-kube-api-access-fq7rl\") pod \"e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") " pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.299488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.299466    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:37.752788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.752762    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"]
Nov 03 12:09:37.754416 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:09:37.754385    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1973fb8_8a5f_458a_8695_b68d91bfea39.slice/crio-45fe5ada09f470bfff630388c0362166083ffff50c89f3acaa00294648457946 WatchSource:0}: Error finding container 45fe5ada09f470bfff630388c0362166083ffff50c89f3acaa00294648457946: Status 404 returned error can't find the container with id 45fe5ada09f470bfff630388c0362166083ffff50c89f3acaa00294648457946
Nov 03 12:09:37.878481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.878446    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" event={"ID":"b1973fb8-8a5f-458a-8695-b68d91bfea39","Type":"ContainerStarted","Data":"34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1"}
Nov 03 12:09:37.878609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.878488    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" event={"ID":"b1973fb8-8a5f-458a-8695-b68d91bfea39","Type":"ContainerStarted","Data":"45fe5ada09f470bfff630388c0362166083ffff50c89f3acaa00294648457946"}
Nov 03 12:09:37.919402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.919373    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:09:37.984200 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:37.984151    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" podStartSLOduration=1.984138475 podStartE2EDuration="1.984138475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:37.981906903 +0000 UTC m=+1126.994373565" watchObservedRunningTime="2026-11-03 12:09:37.984138475 +0000 UTC m=+1126.996605137"
Nov 03 12:09:38.300437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:38.300402    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:39.583924 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:09:39.583885    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:09:47.299893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:47.299855    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:47.312732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:47.312691    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:48.003214 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.003184    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:09:48.859715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.859663    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"]
Nov 03 12:09:48.864099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.864076    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:48.912853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.912822    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:09:48.912853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.912829    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0c5ab4af87ac\""
Nov 03 12:09:48.913021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.912865    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7ab4514d17b9\""
Nov 03 12:09:48.947006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.946977    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-provider-credentials\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:48.947145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.947016    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-plugins-config-volume\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:48.947145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.947057    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-routing-overlay-volume\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:48.947273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.947259    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz5bf\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-kube-api-access-xz5bf\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:48.959005 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.958971    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-763ed2656be0\""
Nov 03 12:09:48.959240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.958973    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-x4r42\""
Nov 03 12:09:48.959240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.958973    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:09:48.959436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.959081    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:09:48.959436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.959338    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:09:48.961768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:48.961744    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"]
Nov 03 12:09:49.048524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.048493    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-provider-credentials\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.048727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.048540    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-plugins-config-volume\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.048727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.048566    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-routing-overlay-volume\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.048866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.048736    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz5bf\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-kube-api-access-xz5bf\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.049352 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.049328    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-routing-overlay-volume\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.049461 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.049391    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-plugins-config-volume\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.051153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.051133    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-provider-credentials\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.101123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.101099    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz5bf\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-kube-api-access-xz5bf\") pod \"payload-processing-external-model-56bf9fbb45-tsq2m\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") " pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.178281 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.178217    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:49.543255 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:49.543213    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"]
Nov 03 12:09:49.545070 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:09:49.545045    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bf6390_7841_4db0_a5a3_3bbcf6ceeb55.slice/crio-4093eb0a7d40c52fc17936a80fcd34133e129edeefcd8056098f531ca8b1fa75 WatchSource:0}: Error finding container 4093eb0a7d40c52fc17936a80fcd34133e129edeefcd8056098f531ca8b1fa75: Status 404 returned error can't find the container with id 4093eb0a7d40c52fc17936a80fcd34133e129edeefcd8056098f531ca8b1fa75
Nov 03 12:09:50.005509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:50.005467    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m" event={"ID":"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55","Type":"ContainerStarted","Data":"a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5"}
Nov 03 12:09:50.005980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:50.005524    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m" event={"ID":"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55","Type":"ContainerStarted","Data":"4093eb0a7d40c52fc17936a80fcd34133e129edeefcd8056098f531ca8b1fa75"}
Nov 03 12:09:50.005980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:50.005544    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:09:50.113176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:50.113127    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m" podStartSLOduration=2.113112839 podStartE2EDuration="2.113112839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:50.109391187 +0000 UTC m=+1139.121857849" watchObservedRunningTime="2026-11-03 12:09:50.113112839 +0000 UTC m=+1139.125579502"
Nov 03 12:09:52.643723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.642594    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"]
Nov 03 12:09:52.667725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.666869    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.695430    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.695493    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ca0240-f3e0-4357-8534-b48e322687a7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.695544    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvj8x\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-kube-api-access-tvj8x\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.695593    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.695629    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.695678    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.695801    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.696091    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ca0240-f3e0-4357-8534-b48e322687a7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.714731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.696120    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.808398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.808361    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ca0240-f3e0-4357-8534-b48e322687a7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.808630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.808613    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.812832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.812803    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.828002 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.813011    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ca0240-f3e0-4357-8534-b48e322687a7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.828339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.809404    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"]
Nov 03 12:09:52.828537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.813748    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.828537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.814414    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ca0240-f3e0-4357-8534-b48e322687a7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.828843 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.828819    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvj8x\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-kube-api-access-tvj8x\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.829693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.829649    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.829871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.829852    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.830417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.830377    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.830575 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.830559    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.832529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.830954    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.833262 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.833196    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.833504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.833486    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.875721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.865684    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.875721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.867209    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ca0240-f3e0-4357-8534-b48e322687a7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.885726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.884223    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:52.913122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:52.907174    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvj8x\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-kube-api-access-tvj8x\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:53.037976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:53.037687    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:53.620779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:53.607590    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"]
Nov 03 12:09:53.695719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:53.691872    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:09:53.695719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:53.691970    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:09:53.695719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:53.692010    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:09:54.097229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:54.096379    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" event={"ID":"64ca0240-f3e0-4357-8534-b48e322687a7","Type":"ContainerStarted","Data":"cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd"}
Nov 03 12:09:54.097229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:54.096444    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" event={"ID":"64ca0240-f3e0-4357-8534-b48e322687a7","Type":"ContainerStarted","Data":"e10809903cc7390474c08c638cce3be034196a5245fad8bed81428092e1d66d5"}
Nov 03 12:09:54.212750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:54.208660    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podStartSLOduration=2.208640854 podStartE2EDuration="2.208640854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:54.20722809 +0000 UTC m=+1143.219694791" watchObservedRunningTime="2026-11-03 12:09:54.208640854 +0000 UTC m=+1143.221107520"
Nov 03 12:09:54.736756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:54.736658    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:54.993886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:54.993788    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" podUID="b1973fb8-8a5f-458a-8695-b68d91bfea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.049926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:55.049896    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:09:55.050298 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:55.050274    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:09:55.050481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:55.050452    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:09:55.064805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:55.064768    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:56.181783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:56.171950    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:09:56.181783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:56.172026    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:09:56.747561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:56.746048    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:57.043777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:57.042745    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:09:57.043777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:57.042815    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:09:57.115930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:57.115791    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:57.150846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:57.150808    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:09:58.058721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:58.050085    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:09:58.058721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:58.050146    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:09:59.050731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:59.050607    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:09:59.050731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:09:59.050666    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:10:00.040107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:00.040067    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:10:00.062918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:00.062799    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:10:01.021984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:01.021952    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:10:01.048911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:01.044068    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:10:01.048911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:01.044125    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:10:02.039548 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:02.039513    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 12:10:02.039731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:02.039565    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 12:10:03.044098 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:03.044067    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:10:03.044512 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:03.044493    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:10:03.045347 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:03.045326    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:10:09.563546 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:10:09.563512    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:10:13.357904 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:13.357867    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:10:17.182863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:17.182827    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:10:17.183516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:17.183465    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:10:36.326056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.326024    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"]
Nov 03 12:10:36.331361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.331339    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.406931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.406898    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f731ab06-b572-43e3-9fef-972fed967e57-tls-certs\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.407099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.406951    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-home\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.407099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.407039    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6bl\" (UniqueName: \"kubernetes.io/projected/f731ab06-b572-43e3-9fef-972fed967e57-kube-api-access-mq6bl\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.407099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.407090    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-dshm\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.407354 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.407327    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-tmp-dir\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.407446 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.407427    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-model-cache\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.426126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.426087    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"distinct-2-w3-bb1b51-kserve-self-signed-certs\""
Nov 03 12:10:36.428688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.428665    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"]
Nov 03 12:10:36.513480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.513439    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-tmp-dir\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.513790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.513765    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-model-cache\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.513955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.513936    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f731ab06-b572-43e3-9fef-972fed967e57-tls-certs\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.514112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.514089    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-home\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.514287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.514270    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6bl\" (UniqueName: \"kubernetes.io/projected/f731ab06-b572-43e3-9fef-972fed967e57-kube-api-access-mq6bl\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.514464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.514448    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-dshm\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.524440 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.521242    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-home\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.524440 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.521531    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-tmp-dir\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.524440 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.521811    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-model-cache\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.526029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.526004    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-dshm\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.540417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.540354    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f731ab06-b572-43e3-9fef-972fed967e57-tls-certs\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.575131 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.575102    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6bl\" (UniqueName: \"kubernetes.io/projected/f731ab06-b572-43e3-9fef-972fed967e57-kube-api-access-mq6bl\") pod \"distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:36.643411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:36.643337    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:37.111598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:37.111574    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"]
Nov 03 12:10:37.113095 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:10:37.113069    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf731ab06_b572_43e3_9fef_972fed967e57.slice/crio-bfc67f3b7e49ed0981db4b2473083a5104a641074f796ce2d7881da073e0aa99 WatchSource:0}: Error finding container bfc67f3b7e49ed0981db4b2473083a5104a641074f796ce2d7881da073e0aa99: Status 404 returned error can't find the container with id bfc67f3b7e49ed0981db4b2473083a5104a641074f796ce2d7881da073e0aa99
Nov 03 12:10:37.613127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:37.613086    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" event={"ID":"f731ab06-b572-43e3-9fef-972fed967e57","Type":"ContainerStarted","Data":"e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1"}
Nov 03 12:10:37.613590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:37.613140    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" event={"ID":"f731ab06-b572-43e3-9fef-972fed967e57","Type":"ContainerStarted","Data":"bfc67f3b7e49ed0981db4b2473083a5104a641074f796ce2d7881da073e0aa99"}
Nov 03 12:10:37.643555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:37.643529    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:37.719552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:37.719499    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podStartSLOduration=1.719481794 podStartE2EDuration="1.719481794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:10:37.717926048 +0000 UTC m=+1186.730392712" watchObservedRunningTime="2026-11-03 12:10:37.719481794 +0000 UTC m=+1186.731948457"
Nov 03 12:10:39.564109 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:10:39.564074    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:10:46.644331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:46.644295    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:46.656596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:46.656569    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:46.676589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:46.676559    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:10:51.261590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:51.261557    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:10:51.262862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:51.262837    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:10:51.722535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:51.722504    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:10:56.761920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:10:56.761890    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"]
Nov 03 12:11:02.759269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:02.759185    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"]
Nov 03 12:11:09.563593 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:11:09.563557    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:11:11.782247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:11.782208    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" podUID="b1973fb8-8a5f-458a-8695-b68d91bfea39" containerName="main" containerID="cri-o://34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1" gracePeriod=45
Nov 03 12:11:12.040346 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.040290    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:11:12.130719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.130679    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1973fb8-8a5f-458a-8695-b68d91bfea39-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1973fb8-8a5f-458a-8695-b68d91bfea39-tls-certs\") pod \"b1973fb8-8a5f-458a-8695-b68d91bfea39\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") "
Nov 03 12:11:12.130858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.130757    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-dshm\") pod \"b1973fb8-8a5f-458a-8695-b68d91bfea39\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") "
Nov 03 12:11:12.130908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.130892    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-home\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-home\") pod \"b1973fb8-8a5f-458a-8695-b68d91bfea39\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") "
Nov 03 12:11:12.130944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.130928    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-model-cache\") pod \"b1973fb8-8a5f-458a-8695-b68d91bfea39\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") "
Nov 03 12:11:12.130990 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.130970    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1973fb8-8a5f-458a-8695-b68d91bfea39-kube-api-access-fq7rl\" (UniqueName: \"kubernetes.io/projected/b1973fb8-8a5f-458a-8695-b68d91bfea39-kube-api-access-fq7rl\") pod \"b1973fb8-8a5f-458a-8695-b68d91bfea39\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") "
Nov 03 12:11:12.131044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.131017    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-tmp-dir\") pod \"b1973fb8-8a5f-458a-8695-b68d91bfea39\" (UID: \"b1973fb8-8a5f-458a-8695-b68d91bfea39\") "
Nov 03 12:11:12.131224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.131184    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-model-cache" pod "b1973fb8-8a5f-458a-8695-b68d91bfea39" (UID: "b1973fb8-8a5f-458a-8695-b68d91bfea39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:12.131224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.131197    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-home" pod "b1973fb8-8a5f-458a-8695-b68d91bfea39" (UID: "b1973fb8-8a5f-458a-8695-b68d91bfea39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:12.131411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.131335    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-tmp-dir" pod "b1973fb8-8a5f-458a-8695-b68d91bfea39" (UID: "b1973fb8-8a5f-458a-8695-b68d91bfea39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:12.131513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.131498    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:12.131555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.131518    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:12.131555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.131535    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:12.132960 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.132940    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1973fb8-8a5f-458a-8695-b68d91bfea39-tls-certs" pod "b1973fb8-8a5f-458a-8695-b68d91bfea39" (UID: "b1973fb8-8a5f-458a-8695-b68d91bfea39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:12.133306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.133288    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-dshm" pod "b1973fb8-8a5f-458a-8695-b68d91bfea39" (UID: "b1973fb8-8a5f-458a-8695-b68d91bfea39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:12.133363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.133299    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1973fb8-8a5f-458a-8695-b68d91bfea39-kube-api-access-fq7rl" pod "b1973fb8-8a5f-458a-8695-b68d91bfea39" (UID: "b1973fb8-8a5f-458a-8695-b68d91bfea39"). InnerVolumeSpecName "kube-api-access-fq7rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:12.232689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.232654    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1973fb8-8a5f-458a-8695-b68d91bfea39-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:12.232689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.232683    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1973fb8-8a5f-458a-8695-b68d91bfea39-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:12.232689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.232720    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq7rl\" (UniqueName: \"kubernetes.io/projected/b1973fb8-8a5f-458a-8695-b68d91bfea39-kube-api-access-fq7rl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:12.860616 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.860587    4909 generic.go:356] "Generic (PLEG): container finished" podID="b1973fb8-8a5f-458a-8695-b68d91bfea39" containerID="34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1" exitCode=0
Nov 03 12:11:12.861091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.860631    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" event={"ID":"b1973fb8-8a5f-458a-8695-b68d91bfea39","Type":"ContainerDied","Data":"34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1"}
Nov 03 12:11:12.861091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.860657    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" event={"ID":"b1973fb8-8a5f-458a-8695-b68d91bfea39","Type":"ContainerDied","Data":"45fe5ada09f470bfff630388c0362166083ffff50c89f3acaa00294648457946"}
Nov 03 12:11:12.861091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.860672    4909 scope.go:122] "RemoveContainer" containerID="34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1"
Nov 03 12:11:12.861091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.860633    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"
Nov 03 12:11:12.877583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.877561    4909 scope.go:122] "RemoveContainer" containerID="34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1"
Nov 03 12:11:12.877897 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:11:12.877877    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1\": container with ID starting with 34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1 not found: ID does not exist" containerID="34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1"
Nov 03 12:11:12.877958 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.877905    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1"} err="failed to get container status \"34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1\": rpc error: code = NotFound desc = could not find container \"34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1\": container with ID starting with 34d06dac6754c9a8d1e52f7c5019dabad2fd1014ede0cfe55ace40cbe96875c1 not found: ID does not exist"
Nov 03 12:11:12.992648 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:12.992615    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2" podUID="b1973fb8-8a5f-458a-8695-b68d91bfea39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:13.045537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:13.045501    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"]
Nov 03 12:11:13.091843 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:13.091817    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-b-78b30cda-kserve-6f68db555f-c6zk2"]
Nov 03 12:11:13.123656 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:13.123593    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1973fb8-8a5f-458a-8695-b68d91bfea39" path="/var/lib/kubelet/pods/b1973fb8-8a5f-458a-8695-b68d91bfea39/volumes"
Nov 03 12:11:17.779227 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:17.779169    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v" podUID="6852b48a-9d35-4116-8d6f-da6ecfb018ea" containerName="main" containerID="cri-o://b538b78baf080032e2350bf277afdbfc9d5b4e9713a161ce18dd7c4713dcb5fc" gracePeriod=45
Nov 03 12:11:17.893221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:17.893195    4909 generic.go:356] "Generic (PLEG): container finished" podID="6852b48a-9d35-4116-8d6f-da6ecfb018ea" containerID="b538b78baf080032e2350bf277afdbfc9d5b4e9713a161ce18dd7c4713dcb5fc" exitCode=0
Nov 03 12:11:17.893339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:17.893267    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v" event={"ID":"6852b48a-9d35-4116-8d6f-da6ecfb018ea","Type":"ContainerDied","Data":"b538b78baf080032e2350bf277afdbfc9d5b4e9713a161ce18dd7c4713dcb5fc"}
Nov 03 12:11:18.038147 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.038087    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:11:18.053171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.053137    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"]
Nov 03 12:11:18.053734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.053691    4909 cpu_manager.go:414] "containerMap: removing container" podUID="6852b48a-9d35-4116-8d6f-da6ecfb018ea" containerName="main"
Nov 03 12:11:18.053734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.053729    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6852b48a-9d35-4116-8d6f-da6ecfb018ea" containerName="main"
Nov 03 12:11:18.053880 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.053770    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b1973fb8-8a5f-458a-8695-b68d91bfea39" containerName="main"
Nov 03 12:11:18.053880 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.053778    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1973fb8-8a5f-458a-8695-b68d91bfea39" containerName="main"
Nov 03 12:11:18.053989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.053948    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="6852b48a-9d35-4116-8d6f-da6ecfb018ea" containerName="main"
Nov 03 12:11:18.053989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.053967    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1973fb8-8a5f-458a-8695-b68d91bfea39" containerName="main"
Nov 03 12:11:18.057403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.057385    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.099422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.099352    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"]
Nov 03 12:11:18.103441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.103423    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.153960 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.153932    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"]
Nov 03 12:11:18.201779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.201750    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-dshm\") pod \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") "
Nov 03 12:11:18.201959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.201799    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tls-certs\") pod \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") "
Nov 03 12:11:18.201959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.201867    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-home\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-home\") pod \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") "
Nov 03 12:11:18.201959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.201950    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6852b48a-9d35-4116-8d6f-da6ecfb018ea-kube-api-access-69x74\" (UniqueName: \"kubernetes.io/projected/6852b48a-9d35-4116-8d6f-da6ecfb018ea-kube-api-access-69x74\") pod \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") "
Nov 03 12:11:18.202194 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202036    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-model-cache\") pod \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") "
Nov 03 12:11:18.202194 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202118    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tmp-dir\") pod \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\" (UID: \"6852b48a-9d35-4116-8d6f-da6ecfb018ea\") "
Nov 03 12:11:18.202370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202334    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d810b-8252-4795-8142-9811c3381eb1-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.202421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202400    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"]
Nov 03 12:11:18.202461 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202421    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202460    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.202495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202491    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d810b-8252-4795-8142-9811c3381eb1-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.202554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202509    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9jl\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-kube-api-access-9h9jl\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.202554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202529    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxwj\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-kube-api-access-rmxwj\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202555    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b851622c-afe9-4f41-9c18-0e04404289ed-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202596    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202628    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202646    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202670    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.202841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202715    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b851622c-afe9-4f41-9c18-0e04404289ed-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202799    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.202942 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.202887    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.203276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.203246    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-model-cache" pod "6852b48a-9d35-4116-8d6f-da6ecfb018ea" (UID: "6852b48a-9d35-4116-8d6f-da6ecfb018ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:18.204072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.204046    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-home" pod "6852b48a-9d35-4116-8d6f-da6ecfb018ea" (UID: "6852b48a-9d35-4116-8d6f-da6ecfb018ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:18.204275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.204255    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tmp-dir" pod "6852b48a-9d35-4116-8d6f-da6ecfb018ea" (UID: "6852b48a-9d35-4116-8d6f-da6ecfb018ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:18.205175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.205146    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.205279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.205217    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.205279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.205247    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.205396 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.205284    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.205396 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.205382    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:18.205493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.205398    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:18.205493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.205413    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:18.206791 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.206752    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-dshm" pod "6852b48a-9d35-4116-8d6f-da6ecfb018ea" (UID: "6852b48a-9d35-4116-8d6f-da6ecfb018ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:11:18.207275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.207247    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tls-certs" pod "6852b48a-9d35-4116-8d6f-da6ecfb018ea" (UID: "6852b48a-9d35-4116-8d6f-da6ecfb018ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:18.207932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.207914    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6852b48a-9d35-4116-8d6f-da6ecfb018ea-kube-api-access-69x74" pod "6852b48a-9d35-4116-8d6f-da6ecfb018ea" (UID: "6852b48a-9d35-4116-8d6f-da6ecfb018ea"). InnerVolumeSpecName "kube-api-access-69x74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:18.306765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306657    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.306765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306735    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b851622c-afe9-4f41-9c18-0e04404289ed-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.306969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306771    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.306969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306797    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.306969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306845    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.306969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306890    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.306969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306918    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.306969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.306957    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.307266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307013    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d810b-8252-4795-8142-9811c3381eb1-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307095    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.307266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307143    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307158    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307208    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d810b-8252-4795-8142-9811c3381eb1-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307236    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307246    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9jl\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-kube-api-access-9h9jl\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307286    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307310    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxwj\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-kube-api-access-rmxwj\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307355    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b851622c-afe9-4f41-9c18-0e04404289ed-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307359    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307428    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307456    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.307563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307473    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.308072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307813    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.308072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307865    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.308072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.307928    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.308072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.308006    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6852b48a-9d35-4116-8d6f-da6ecfb018ea-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:18.308072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.308022    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6852b48a-9d35-4116-8d6f-da6ecfb018ea-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:18.308072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.308021    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d810b-8252-4795-8142-9811c3381eb1-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.308072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.308038    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69x74\" (UniqueName: \"kubernetes.io/projected/6852b48a-9d35-4116-8d6f-da6ecfb018ea-kube-api-access-69x74\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:18.308416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.308188    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b851622c-afe9-4f41-9c18-0e04404289ed-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.308416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.308244    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.309653 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.309632    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.309918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.309900    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.309993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.309908    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b851622c-afe9-4f41-9c18-0e04404289ed-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.309993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.309941    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d810b-8252-4795-8142-9811c3381eb1-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.359170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.359137    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.359296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.359226    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9jl\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-kube-api-access-9h9jl\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.359366 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.359298    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.359973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.359954    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxwj\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-kube-api-access-rmxwj\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.370379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.370361    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:18.416984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.416954    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:18.787663 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.787631    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"]
Nov 03 12:11:18.825592 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:11:18.825563    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20d810b_8252_4795_8142_9811c3381eb1.slice/crio-b31a43be6cc53d952b0194b48f34c6b3b977215511eab099c70833728d6b0e22 WatchSource:0}: Error finding container b31a43be6cc53d952b0194b48f34c6b3b977215511eab099c70833728d6b0e22: Status 404 returned error can't find the container with id b31a43be6cc53d952b0194b48f34c6b3b977215511eab099c70833728d6b0e22
Nov 03 12:11:18.828151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.828117    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:18.828274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.828196    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:18.828274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.828239    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:18.837386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.837360    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"]
Nov 03 12:11:18.840771 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:11:18.840749    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb851622c_afe9_4f41_9c18_0e04404289ed.slice/crio-e8169f70069f61015afff0cbd5a32e90f77480b818fa3053a198a55f7f61129f WatchSource:0}: Error finding container e8169f70069f61015afff0cbd5a32e90f77480b818fa3053a198a55f7f61129f: Status 404 returned error can't find the container with id e8169f70069f61015afff0cbd5a32e90f77480b818fa3053a198a55f7f61129f
Nov 03 12:11:18.842955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.842927    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:18.843062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.842994    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:18.843062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.843038    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:18.901018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.900987    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" event={"ID":"b851622c-afe9-4f41-9c18-0e04404289ed","Type":"ContainerStarted","Data":"e8169f70069f61015afff0cbd5a32e90f77480b818fa3053a198a55f7f61129f"}
Nov 03 12:11:18.902204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.902181    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" event={"ID":"d20d810b-8252-4795-8142-9811c3381eb1","Type":"ContainerStarted","Data":"b31a43be6cc53d952b0194b48f34c6b3b977215511eab099c70833728d6b0e22"}
Nov 03 12:11:18.903761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.903744    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"
Nov 03 12:11:18.903761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.903748    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v" event={"ID":"6852b48a-9d35-4116-8d6f-da6ecfb018ea","Type":"ContainerDied","Data":"b1c7828be955d33bcd59fbb08db4f1baeeec6950fde009773420e49ee385ae46"}
Nov 03 12:11:18.903926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:18.903791    4909 scope.go:122] "RemoveContainer" containerID="b538b78baf080032e2350bf277afdbfc9d5b4e9713a161ce18dd7c4713dcb5fc"
Nov 03 12:11:19.056600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:19.056522    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"]
Nov 03 12:11:19.059000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:19.058977    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/e2e-conflict-a-78b30cda-kserve-59d5556b4f-jz66v"]
Nov 03 12:11:19.124828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:19.124797    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6852b48a-9d35-4116-8d6f-da6ecfb018ea" path="/var/lib/kubelet/pods/6852b48a-9d35-4116-8d6f-da6ecfb018ea/volumes"
Nov 03 12:11:19.911813 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:19.911778    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" event={"ID":"b851622c-afe9-4f41-9c18-0e04404289ed","Type":"ContainerStarted","Data":"28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402"}
Nov 03 12:11:19.913819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:19.913795    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" event={"ID":"d20d810b-8252-4795-8142-9811c3381eb1","Type":"ContainerStarted","Data":"542633bd06ff738e7bbe8ee7828430ba2cee52919a4eb9d9da216af239ac1f6c"}
Nov 03 12:11:20.025409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.025355    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podStartSLOduration=2.025341133 podStartE2EDuration="2.025341133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:20.022773802 +0000 UTC m=+1229.035240465" watchObservedRunningTime="2026-11-03 12:11:20.025341133 +0000 UTC m=+1229.037807795"
Nov 03 12:11:20.134444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.134393    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podStartSLOduration=2.134377588 podStartE2EDuration="2.134377588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:20.133319286 +0000 UTC m=+1229.145785950" watchObservedRunningTime="2026-11-03 12:11:20.134377588 +0000 UTC m=+1229.146844242"
Nov 03 12:11:20.375066 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.375035    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:20.379216 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.379188    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:11:20.379463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.379433    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 12:11:20.417655 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.417622    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:20.419029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.419003    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:11:20.419151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:20.419047    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 12:11:21.374721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:21.373060    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:11:21.374721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:21.373123    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 12:11:21.417761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:21.417732    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:11:21.417975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:21.417952    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 12:11:22.373728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:22.373071    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:11:22.373728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:22.373135    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 12:11:22.417540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:22.417508    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:11:22.417928 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:22.417555    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 12:11:23.378732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:23.378053    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:11:23.378732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:23.378118    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 12:11:23.418055 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:23.418026    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:11:23.418511 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:23.418486    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 12:11:24.384721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:24.371878    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:11:24.384721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:24.371940    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 12:11:24.418190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:24.418158    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:11:24.418574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:24.418207    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 12:11:25.371007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:25.370972    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:11:25.371186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:25.371036    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 12:11:25.417549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:25.417513    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:11:25.417678 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:25.417572    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 12:11:25.842371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:25.842335    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:11:26.375409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:26.375377    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:26.375769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:26.375743    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:26.376463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:26.376445    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:11:26.632629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:26.630889    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:26.632629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:26.631454    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:26.655806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:26.642342    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:11:27.966724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:27.966486    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:28.173720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:28.172779    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.362719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:29.362612    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:11:29.362719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:29.362712    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:32.930756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:32.930720    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-sbjdh-vzpcf"]
Nov 03 12:11:32.935903 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:32.935886    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:32.986085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:32.986052    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sbjdh-vzpcf"]
Nov 03 12:11:33.020892 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.020858    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-trusted-ca-bundle\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.021037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.021008    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6vg\" (UniqueName: \"kubernetes.io/projected/9a33ef25-e91e-44b6-8644-4fb496d9e632-kube-api-access-4v6vg\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.021161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.021138    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-service-ca-bundle\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.021289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.021198    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a33ef25-e91e-44b6-8644-4fb496d9e632-archives-path\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.028675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.028657    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pf74h\""
Nov 03 12:11:33.122819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.122795    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a33ef25-e91e-44b6-8644-4fb496d9e632-archives-path\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.122977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.122899    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-trusted-ca-bundle\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.122977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.122966    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6vg\" (UniqueName: \"kubernetes.io/projected/9a33ef25-e91e-44b6-8644-4fb496d9e632-kube-api-access-4v6vg\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.123065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.123038    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-service-ca-bundle\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.123228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.123209    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a33ef25-e91e-44b6-8644-4fb496d9e632-archives-path\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.123757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.123732    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-service-ca-bundle\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.123809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.123767    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-trusted-ca-bundle\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.176215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.176190    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6vg\" (UniqueName: \"kubernetes.io/projected/9a33ef25-e91e-44b6-8644-4fb496d9e632-kube-api-access-4v6vg\") pod \"periodic-gathering-sbjdh-vzpcf\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") " pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.178939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.178915    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"]
Nov 03 12:11:33.183097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.183050    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224332    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0025920d-a5a7-481f-97b3-84e86203ad72-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224385    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2rs\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-kube-api-access-nf2rs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224472    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224682    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224734    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224780    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0025920d-a5a7-481f-97b3-84e86203ad72-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224842    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.224988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.224954    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.225042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.225018    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.247961 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.247934    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:11:33.265973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.265949    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"]
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.326593    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0025920d-a5a7-481f-97b3-84e86203ad72-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.326639    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.326691    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.326776    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.326844    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0025920d-a5a7-481f-97b3-84e86203ad72-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.326881    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2rs\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-kube-api-access-nf2rs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.326916    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.327076    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.327101    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.328251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.327541    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.331939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.331916    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.332230 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.332211    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0025920d-a5a7-481f-97b3-84e86203ad72-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.332431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.332413    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.332623 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.332604    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.333797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.333767    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.339290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.339244    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0025920d-a5a7-481f-97b3-84e86203ad72-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.382420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.382384    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.382535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.382437    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2rs\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-kube-api-access-nf2rs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.500153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.500118    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:33.628485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.628448    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sbjdh-vzpcf"]
Nov 03 12:11:33.631857 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:11:33.631825    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a33ef25_e91e_44b6_8644_4fb496d9e632.slice/crio-b6fc7a4040b31955c4222b577f308c79cf08e09aac830dac57f588fd59a792f4 WatchSource:0}: Error finding container b6fc7a4040b31955c4222b577f308c79cf08e09aac830dac57f588fd59a792f4: Status 404 returned error can't find the container with id b6fc7a4040b31955c4222b577f308c79cf08e09aac830dac57f588fd59a792f4
Nov 03 12:11:33.871248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.871214    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"]
Nov 03 12:11:33.872388 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:11:33.872358    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0025920d_a5a7_481f_97b3_84e86203ad72.slice/crio-3e40d8d7849f37df4951f2d354804e2ab92578e0ea7240f60aa495295b8d37a6 WatchSource:0}: Error finding container 3e40d8d7849f37df4951f2d354804e2ab92578e0ea7240f60aa495295b8d37a6: Status 404 returned error can't find the container with id 3e40d8d7849f37df4951f2d354804e2ab92578e0ea7240f60aa495295b8d37a6
Nov 03 12:11:33.874751 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.874721    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:33.874824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.874785    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:33.874824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:33.874815    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:11:34.299008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.298973    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" event={"ID":"0025920d-a5a7-481f-97b3-84e86203ad72","Type":"ContainerStarted","Data":"466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088"}
Nov 03 12:11:34.299495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.299029    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" event={"ID":"0025920d-a5a7-481f-97b3-84e86203ad72","Type":"ContainerStarted","Data":"3e40d8d7849f37df4951f2d354804e2ab92578e0ea7240f60aa495295b8d37a6"}
Nov 03 12:11:34.300615 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.300590    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf" event={"ID":"9a33ef25-e91e-44b6-8644-4fb496d9e632","Type":"ContainerStarted","Data":"4d5e933e7b7871a706156224e54c71b6dc85f7141926f2309c6b605c944ca66b"}
Nov 03 12:11:34.300749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.300627    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf" event={"ID":"9a33ef25-e91e-44b6-8644-4fb496d9e632","Type":"ContainerStarted","Data":"b6fc7a4040b31955c4222b577f308c79cf08e09aac830dac57f588fd59a792f4"}
Nov 03 12:11:34.409243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.409180    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podStartSLOduration=1.409160041 podStartE2EDuration="1.409160041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:34.407462451 +0000 UTC m=+1243.419929114" watchObservedRunningTime="2026-11-03 12:11:34.409160041 +0000 UTC m=+1243.421626705"
Nov 03 12:11:34.500635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.500601    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:34.502078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.502047    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:11:34.502188 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.502097    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 12:11:34.516584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:34.516542    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf" podStartSLOduration=2.516529578 podStartE2EDuration="2.516529578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:34.514776798 +0000 UTC m=+1243.527243458" watchObservedRunningTime="2026-11-03 12:11:34.516529578 +0000 UTC m=+1243.528996241"
Nov 03 12:11:35.500646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:35.500575    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:11:35.506026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:35.501260    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 12:11:36.501005 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:36.500962    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:11:36.501406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:36.501020    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 12:11:37.500978 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:37.500944    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:11:37.501138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:37.500996    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 12:11:38.501299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:38.501264    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:11:38.501750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:38.501316    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 12:11:39.506081 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:39.506044    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:11:39.507109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:39.507074    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 12:11:39.608631 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:11:39.608594    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:11:40.503532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:40.503082    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:11:40.503532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:40.503148    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 12:11:41.667809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:41.667765    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:11:41.735721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:41.735271    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 12:11:41.870876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:41.870817    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 03 12:11:42.165811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:42.165767    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Nov 03 12:11:42.501380 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:42.501339    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:11:42.501568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:42.501406    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:43.542869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:43.542831    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:11:43.543279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:43.542894    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:43.604796 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:43.604764    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:43.605272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:43.605244    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:43.605971 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:43.605954    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:11:54.564555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.564522    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"]
Nov 03 12:11:54.570102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.570081    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.612092 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.612070    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-dshm\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.612198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.612096    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-model-cache\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.612237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.612223    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tmp-dir\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.612276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.612253    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-home\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.612306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.612291    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dqp\" (UniqueName: \"kubernetes.io/projected/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-kube-api-access-b4dqp\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.612369 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.612352    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tls-certs\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.620443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.620422    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"openshift-service-ca.crt\""
Nov 03 12:11:54.620576 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.620537    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"kube-root-ca.crt\""
Nov 03 12:11:54.666842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.666809    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"distinct-w4-8d2c7c-kserve-self-signed-certs\""
Nov 03 12:11:54.667052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.666932    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"default-dockercfg-n4q48\""
Nov 03 12:11:54.671771 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.671745    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"]
Nov 03 12:11:54.713195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713170    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tmp-dir\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713215    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-home\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713254    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dqp\" (UniqueName: \"kubernetes.io/projected/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-kube-api-access-b4dqp\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713434 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713387    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tls-certs\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713530    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-dshm\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713556    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-model-cache\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713576    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tmp-dir\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713610    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-home\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.713963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.713941    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-model-cache\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.715710 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.715673    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-dshm\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.716017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.715998    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tls-certs\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.779007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.778938    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dqp\" (UniqueName: \"kubernetes.io/projected/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-kube-api-access-b4dqp\") pod \"distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:54.882484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:54.882401    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:55.250300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:55.250277    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"]
Nov 03 12:11:55.252322 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:11:55.252297    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb0a58e5_d12c_4e1e_8015_925a4d85ed35.slice/crio-19a02d2120c639d2fdab89427035d2017c991e6c9d1de3ec94909bdb7303607f WatchSource:0}: Error finding container 19a02d2120c639d2fdab89427035d2017c991e6c9d1de3ec94909bdb7303607f: Status 404 returned error can't find the container with id 19a02d2120c639d2fdab89427035d2017c991e6c9d1de3ec94909bdb7303607f
Nov 03 12:11:55.518284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:55.518191    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" event={"ID":"fb0a58e5-d12c-4e1e-8015-925a4d85ed35","Type":"ContainerStarted","Data":"2e20e1bdb062325531acc08d346f48878e57184f84a64a71b2792084c719211c"}
Nov 03 12:11:55.518284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:55.518246    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" event={"ID":"fb0a58e5-d12c-4e1e-8015-925a4d85ed35","Type":"ContainerStarted","Data":"19a02d2120c639d2fdab89427035d2017c991e6c9d1de3ec94909bdb7303607f"}
Nov 03 12:11:55.626256 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:55.626208    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podStartSLOduration=1.626192957 podStartE2EDuration="1.626192957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:55.62334905 +0000 UTC m=+1264.635815716" watchObservedRunningTime="2026-11-03 12:11:55.626192957 +0000 UTC m=+1264.638659619"
Nov 03 12:11:55.882943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:55.882857    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:11:59.420255 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:11:59.420220    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:12:04.883214 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:04.883179    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:12:04.897141 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:04.897115    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:12:05.730606 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:05.730562    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:12:09.564831 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:12:09.564797    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:12:20.676076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:20.674796    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.731818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:20.731769    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.850670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:20.850627    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 03 12:12:21.069722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.068823    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.354777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.353781    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.734862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.734810    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.772877 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.772831    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.836137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.798290    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.912750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.866911    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.912750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.875029    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:12:21.912750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.904836    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:12:21.912750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:21.904888    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:12:22.148709 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:22.133795    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:12:22.148709 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:22.134170    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:12:22.149825 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:22.149779    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:22.405324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:22.405232    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 03 12:12:22.419731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:22.419682    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:12:22.907396 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:22.907337    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:22.916099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:22.916061    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:12:23.026722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.015050    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:23.037434 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.036014    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 12:12:23.039185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.039162    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:12:23.123385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.123354    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:12:23.144515 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.144486    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:12:23.157791 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.157618    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:23.235148 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.235115    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:12:23.302460 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.302431    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:12:23.376858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.376042    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:12:23.379296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:23.379270    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:12:31.238862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.238816    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"]
Nov 03 12:12:31.244471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.244420    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.296623 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.296583    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"]
Nov 03 12:12:31.338397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.338369    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"distinct-2-w4-8d2c7c-kserve-self-signed-certs\""
Nov 03 12:12:31.391724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.389094    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-dshm\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.391724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.389171    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-home\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.391724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.389208    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47bm6\" (UniqueName: \"kubernetes.io/projected/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-kube-api-access-47bm6\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.391724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.389415    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tls-certs\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.391724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.389452    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-model-cache\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.391724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.389527    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tmp-dir\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.490688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.490599    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-dshm\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.490688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.490657    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-home\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.490688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.490690    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47bm6\" (UniqueName: \"kubernetes.io/projected/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-kube-api-access-47bm6\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.491024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.490849    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tls-certs\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.491024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.490881    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-model-cache\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.491024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.490953    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tmp-dir\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.491188 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.491154    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-home\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.491334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.491217    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-model-cache\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.491405 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.491384    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tmp-dir\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.492945 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.492924    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-dshm\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.493454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.493431    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tls-certs\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.543129 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.543104    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47bm6\" (UniqueName: \"kubernetes.io/projected/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-kube-api-access-47bm6\") pod \"distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") " pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:31.557006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:31.556985    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:32.017245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:32.017218    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"]
Nov 03 12:12:32.019571 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:12:32.019546    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4cb2d07_e297_42a6_bf7e_5b28b1d4d2f2.slice/crio-3cb64b8a2aa9eb2600cb71f674846f8bf16001a4f9d4e22259b9318714a56471 WatchSource:0}: Error finding container 3cb64b8a2aa9eb2600cb71f674846f8bf16001a4f9d4e22259b9318714a56471: Status 404 returned error can't find the container with id 3cb64b8a2aa9eb2600cb71f674846f8bf16001a4f9d4e22259b9318714a56471
Nov 03 12:12:32.298950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:32.298858    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" event={"ID":"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2","Type":"ContainerStarted","Data":"f7b6a7b6f12d83ebeb4b60a1e70a7f2aae796749cc3d606e6fd206e4519e30e0"}
Nov 03 12:12:32.298950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:32.298907    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" event={"ID":"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2","Type":"ContainerStarted","Data":"3cb64b8a2aa9eb2600cb71f674846f8bf16001a4f9d4e22259b9318714a56471"}
Nov 03 12:12:32.406873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:32.405899    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podStartSLOduration=1.405872809 podStartE2EDuration="1.405872809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:32.404415395 +0000 UTC m=+1301.416882056" watchObservedRunningTime="2026-11-03 12:12:32.405872809 +0000 UTC m=+1301.418339474"
Nov 03 12:12:32.572330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:32.572237    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:38.931109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:38.931071    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:12:39.635691 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:12:39.635645    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:12:41.208468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:41.208423    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.239832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:41.239784    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.388835 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:41.388776    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.566874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:41.566785    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:41.966784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:41.947800    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.968890 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:41.968839    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.042850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.042814    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:42.043138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.043108    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.084844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.084811    4909 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:42.085113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.085077    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.162984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.162942    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.163597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.163215    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.231898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.231790    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.232881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.232858    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:12:42.271727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.257748    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 12:12:42.311657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.311623    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:12:42.542775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.540406    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.542775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.542336    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:12:42.696724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.636435    4909 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Nov 03 12:12:42.971721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.963061    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.971721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:42.963391    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:12:43.054743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.053966    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.072724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.056083    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:12:43.114722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.113806    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:43.114722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.113889    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:43.196723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.195817    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:43.319781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.295522    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.395723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.349456    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:43.543838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.543790    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.700783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.689565    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:12:43.915723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.912491    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Nov 03 12:12:44.008821 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:43.998800    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:44.076748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:44.056794    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:44.123950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:44.123916    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:12:44.148848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:44.148812    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:12:44.197931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:44.173502    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:12:44.197931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:44.175861    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:12:44.197931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:44.189821    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:12:44.197931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:44.190043    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:12:51.569746 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:51.569712    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:12:51.580790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:12:51.580765    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:13:02.525721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.524582    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"]
Nov 03 12:13:02.544095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.544067    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:02.599383 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.599095    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdrb\" (UniqueName: \"kubernetes.io/projected/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-kube-api-access-6hdrb\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:02.599383 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.599226    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:02.615794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.614952    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-69fd8043\""
Nov 03 12:13:02.615794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.615155    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-69fd8043-dockercfg-66kpw\""
Nov 03 12:13:02.627907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.627855    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"]
Nov 03 12:13:02.749744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.735913    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"]
Nov 03 12:13:02.761202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.761160    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdrb\" (UniqueName: \"kubernetes.io/projected/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-kube-api-access-6hdrb\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:02.761386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.761338    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:02.762427 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.762395    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:02.781854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.781062    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"]
Nov 03 12:13:02.781854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.781220    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:02.814867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.814842    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdrb\" (UniqueName: \"kubernetes.io/projected/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-kube-api-access-6hdrb\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:02.964272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.964237    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqnt4\" (UniqueName: \"kubernetes.io/projected/4e20ef1f-cf79-4c2d-9374-90106a25e594-kube-api-access-zqnt4\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:02.964452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:02.964373    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e20ef1f-cf79-4c2d-9374-90106a25e594-plugins-config-volume\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:03.059617 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.059548    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:03.065544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.065514    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqnt4\" (UniqueName: \"kubernetes.io/projected/4e20ef1f-cf79-4c2d-9374-90106a25e594-kube-api-access-zqnt4\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:03.065668 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.065594    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e20ef1f-cf79-4c2d-9374-90106a25e594-plugins-config-volume\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:03.066484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.066458    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e20ef1f-cf79-4c2d-9374-90106a25e594-plugins-config-volume\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:03.117916 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.117890    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqnt4\" (UniqueName: \"kubernetes.io/projected/4e20ef1f-cf79-4c2d-9374-90106a25e594-kube-api-access-zqnt4\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:03.403027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.402951    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:03.427519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.427493    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"]
Nov 03 12:13:03.428468 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:13:03.428445    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1daa27_fbe3_4b6e_b32f_52fe3017e1a3.slice/crio-991814c8b4566469d2defb69798e99728bf7f88e6b0a0e49705ef58b57d5ee0d WatchSource:0}: Error finding container 991814c8b4566469d2defb69798e99728bf7f88e6b0a0e49705ef58b57d5ee0d: Status 404 returned error can't find the container with id 991814c8b4566469d2defb69798e99728bf7f88e6b0a0e49705ef58b57d5ee0d
Nov 03 12:13:03.770865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:03.770838    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"]
Nov 03 12:13:03.771976 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:13:03.771950    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e20ef1f_cf79_4c2d_9374_90106a25e594.slice/crio-f7cf673060a59bdb21a3c8023188445af430d04661dac247393c5ac6954d9cd6 WatchSource:0}: Error finding container f7cf673060a59bdb21a3c8023188445af430d04661dac247393c5ac6954d9cd6: Status 404 returned error can't find the container with id f7cf673060a59bdb21a3c8023188445af430d04661dac247393c5ac6954d9cd6
Nov 03 12:13:04.447557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.439545    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" event={"ID":"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3","Type":"ContainerStarted","Data":"1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b"}
Nov 03 12:13:04.447557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.439632    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" event={"ID":"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3","Type":"ContainerStarted","Data":"991814c8b4566469d2defb69798e99728bf7f88e6b0a0e49705ef58b57d5ee0d"}
Nov 03 12:13:04.447557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.440678    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:04.469717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.458734    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" event={"ID":"4e20ef1f-cf79-4c2d-9374-90106a25e594","Type":"ContainerStarted","Data":"8fbab4255081e28ee679662edfd33f05e54b821c953437d230a4a455e5da52c3"}
Nov 03 12:13:04.469717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.458803    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" event={"ID":"4e20ef1f-cf79-4c2d-9374-90106a25e594","Type":"ContainerStarted","Data":"f7cf673060a59bdb21a3c8023188445af430d04661dac247393c5ac6954d9cd6"}
Nov 03 12:13:04.469717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.458832    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:04.555720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.555381    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" podStartSLOduration=2.555366164 podStartE2EDuration="2.555366164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:04.551066481 +0000 UTC m=+1333.563533144" watchObservedRunningTime="2026-11-03 12:13:04.555366164 +0000 UTC m=+1333.567832870"
Nov 03 12:13:04.658721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:04.657977    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" podStartSLOduration=2.6579567170000002 podStartE2EDuration="2.657956717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:04.656530499 +0000 UTC m=+1333.668997163" watchObservedRunningTime="2026-11-03 12:13:04.657956717 +0000 UTC m=+1333.670423381"
Nov 03 12:13:07.102552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.102516    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"]
Nov 03 12:13:07.108928 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.108892    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177123    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177178    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177196    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177420    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177459    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpskc\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-kube-api-access-dpskc\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177523    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177611    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177733    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.177841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.177829    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.207008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.206984    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"]
Nov 03 12:13:07.278468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278440    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278477    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278527    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278562    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpskc\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-kube-api-access-dpskc\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278603    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278655    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278739    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278801    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.278885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278869    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.279099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.278887    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.279099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.279085    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.279204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.279169    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.279204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.279190    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.279315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.279290    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.280804 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.280782    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.281301 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.281279    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.331020 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.330991    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.331446 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.331006    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpskc\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-kube-api-access-dpskc\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.427086 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.427008    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:07.840861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.840825    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"]
Nov 03 12:13:07.877570 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:13:07.877541    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090e61b1_c21b_4fe9_9ca0_7e536abdd75b.slice/crio-435cc39f6b32f05809cd857fa4909f296026fc9f390fdde3076a69d348db3451 WatchSource:0}: Error finding container 435cc39f6b32f05809cd857fa4909f296026fc9f390fdde3076a69d348db3451: Status 404 returned error can't find the container with id 435cc39f6b32f05809cd857fa4909f296026fc9f390fdde3076a69d348db3451
Nov 03 12:13:07.880152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.880117    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:13:07.880237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.880186    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:13:07.880237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:07.880216    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:13:08.552040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:08.552005    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" event={"ID":"090e61b1-c21b-4fe9-9ca0-7e536abdd75b","Type":"ContainerStarted","Data":"090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0"}
Nov 03 12:13:08.552418 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:08.552068    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" event={"ID":"090e61b1-c21b-4fe9-9ca0-7e536abdd75b","Type":"ContainerStarted","Data":"435cc39f6b32f05809cd857fa4909f296026fc9f390fdde3076a69d348db3451"}
Nov 03 12:13:08.760720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:08.754942    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podStartSLOduration=1.7549213350000001 podStartE2EDuration="1.754921335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:08.705957031 +0000 UTC m=+1337.718423693" watchObservedRunningTime="2026-11-03 12:13:08.754921335 +0000 UTC m=+1337.767388000"
Nov 03 12:13:09.428121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:09.428085    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:09.429253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:09.429228    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:13:09.429386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:09.429283    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 12:13:09.591913 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:13:09.591877    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:13:10.428210 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:10.428171    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:13:10.428380 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:10.428230    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 12:13:10.588566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:10.588535    4909 generic.go:356] "Generic (PLEG): container finished" podID="9a33ef25-e91e-44b6-8644-4fb496d9e632" containerID="4d5e933e7b7871a706156224e54c71b6dc85f7141926f2309c6b605c944ca66b" exitCode=0
Nov 03 12:13:10.588717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:10.588599    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf" event={"ID":"9a33ef25-e91e-44b6-8644-4fb496d9e632","Type":"ContainerDied","Data":"4d5e933e7b7871a706156224e54c71b6dc85f7141926f2309c6b605c944ca66b"}
Nov 03 12:13:11.427890 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.427853    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:13:11.428358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.427909    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 12:13:11.802170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.802144    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:13:11.835057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835031    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a33ef25-e91e-44b6-8644-4fb496d9e632-kube-api-access-4v6vg\" (UniqueName: \"kubernetes.io/projected/9a33ef25-e91e-44b6-8644-4fb496d9e632-kube-api-access-4v6vg\") pod \"9a33ef25-e91e-44b6-8644-4fb496d9e632\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") "
Nov 03 12:13:11.835195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835181    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-service-ca-bundle\") pod \"9a33ef25-e91e-44b6-8644-4fb496d9e632\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") "
Nov 03 12:13:11.835248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835214    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-trusted-ca-bundle\") pod \"9a33ef25-e91e-44b6-8644-4fb496d9e632\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") "
Nov 03 12:13:11.835336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835324    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a33ef25-e91e-44b6-8644-4fb496d9e632-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a33ef25-e91e-44b6-8644-4fb496d9e632-archives-path\") pod \"9a33ef25-e91e-44b6-8644-4fb496d9e632\" (UID: \"9a33ef25-e91e-44b6-8644-4fb496d9e632\") "
Nov 03 12:13:11.835532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835505    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-service-ca-bundle" pod "9a33ef25-e91e-44b6-8644-4fb496d9e632" (UID: "9a33ef25-e91e-44b6-8644-4fb496d9e632"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:11.835659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835637    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-trusted-ca-bundle" pod "9a33ef25-e91e-44b6-8644-4fb496d9e632" (UID: "9a33ef25-e91e-44b6-8644-4fb496d9e632"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:11.835739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835723    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a33ef25-e91e-44b6-8644-4fb496d9e632-archives-path" pod "9a33ef25-e91e-44b6-8644-4fb496d9e632" (UID: "9a33ef25-e91e-44b6-8644-4fb496d9e632"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:11.835803 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835746    4909 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-service-ca-bundle\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:11.835803 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.835760    4909 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a33ef25-e91e-44b6-8644-4fb496d9e632-trusted-ca-bundle\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:11.837269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.837248    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a33ef25-e91e-44b6-8644-4fb496d9e632-kube-api-access-4v6vg" pod "9a33ef25-e91e-44b6-8644-4fb496d9e632" (UID: "9a33ef25-e91e-44b6-8644-4fb496d9e632"). InnerVolumeSpecName "kube-api-access-4v6vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:11.937531 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.937499    4909 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a33ef25-e91e-44b6-8644-4fb496d9e632-archives-path\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:11.937660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:11.937536    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6vg\" (UniqueName: \"kubernetes.io/projected/9a33ef25-e91e-44b6-8644-4fb496d9e632-kube-api-access-4v6vg\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:12.428290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:12.428252    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:13:12.428742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:12.428306    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 12:13:12.601712 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:12.601674    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf" event={"ID":"9a33ef25-e91e-44b6-8644-4fb496d9e632","Type":"ContainerDied","Data":"b6fc7a4040b31955c4222b577f308c79cf08e09aac830dac57f588fd59a792f4"}
Nov 03 12:13:12.601875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:12.601749    4909 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6fc7a4040b31955c4222b577f308c79cf08e09aac830dac57f588fd59a792f4"
Nov 03 12:13:12.601875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:12.601681    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sbjdh-vzpcf"
Nov 03 12:13:13.427546 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:13.427510    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:13:13.427769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:13.427572    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 12:13:14.428406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:14.428368    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:13:14.428942 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:14.428424    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 12:13:15.432968 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:15.432930    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:15.433419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:15.433233    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:15.434042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:15.434022    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:13:15.536752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:15.536723    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:13:16.541355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:16.541329    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:13:22.751883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:22.751848    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:13:39.577230 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:13:39.577187    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:13:41.340752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.335756    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.665777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.665677    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"]
Nov 03 12:13:41.667170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.667140    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9a33ef25-e91e-44b6-8644-4fb496d9e632" containerName="insights-gathering"
Nov 03 12:13:41.667334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.667317    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a33ef25-e91e-44b6-8644-4fb496d9e632" containerName="insights-gathering"
Nov 03 12:13:41.667629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.667613    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a33ef25-e91e-44b6-8644-4fb496d9e632" containerName="insights-gathering"
Nov 03 12:13:41.676577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.676541    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 12:13:41.703596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.701602    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:13:41.707608 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.707580    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"]
Nov 03 12:13:41.707897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.707879    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.780474    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-data\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.780616    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-socket\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.780735    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-credential-socket\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.780828    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cb4163-c911-4f2b-97f8-4643635d95c7-istiod-ca-cert\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.780906    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-certs\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.780961    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-podinfo\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.780990    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-envoy\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.781016    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-token\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.782203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.781062    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-kube-api-access-dmrq5\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:41.813022 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:41.812989    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ac6071-openshift-default-dockercfg-brkhz\""
Nov 03 12:13:42.048641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.048606    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cb4163-c911-4f2b-97f8-4643635d95c7-istiod-ca-cert\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.053076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.053049    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-certs\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.084297 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.053434    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-podinfo\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.085042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.084619    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-envoy\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.085222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.085204    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-token\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.086052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.086031    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-kube-api-access-dmrq5\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.086810 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.086616    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-data\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.087415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.087396    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-socket\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.090068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.090010    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-data\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.090481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.090434    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-socket\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.126528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.126484    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-certs\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.127035 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.127007    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-credential-socket\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.131924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.131900    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cb4163-c911-4f2b-97f8-4643635d95c7-istiod-ca-cert\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.157745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.154450    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-credential-socket\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.167727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.158369    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.167727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.159868    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.212722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.207337    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-envoy\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.213441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.213384    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-token\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.241734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.230318    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-podinfo\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.358778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.322049    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-kube-api-access-dmrq5\") pod \"e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") " pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.374272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.374238    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:42.687806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.652786    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Nov 03 12:13:42.687806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.653093    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:13:42.687806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.653424    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 12:13:42.687806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.653682    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:13:42.687806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.654005    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:13:42.734757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.734245    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.734757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:42.734594    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:13:43.102755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.088291    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.184666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.183887    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.361772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.361469    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:13:43.608734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.592801    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 12:13:43.670344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.653792    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.670344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.658136    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.670344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.664769    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:13:43.670344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.665832    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:13:43.670344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.665893    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:13:43.755755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.729553    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:13:43.755755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.729647    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:13:43.755755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.729715    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:13:43.755755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.735362    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:13:43.797274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.797239    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"]
Nov 03 12:13:44.007108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:43.977833    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Nov 03 12:13:44.166848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.166802    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.168140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.168118    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:13:44.192722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.186188    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.305402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.305347    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:44.328241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.328192    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.346292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.346262    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:13:44.392867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.392831    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:44.393453 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.393421    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.752832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.672941    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 12:13:44.752832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.673369    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:13:44.752832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.697398    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.752832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.700409    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 12:13:44.752832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:44.739767    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:13:45.122638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.122541    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" event={"ID":"d8cb4163-c911-4f2b-97f8-4643635d95c7","Type":"ContainerStarted","Data":"3cf2237fe2cf4f10d2c2cfcde2f4bdd778a71c8ff2ca5b3241e2ad4127254628"}
Nov 03 12:13:45.193164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.193121    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.214728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.210768    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.683724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.682000    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.703793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.700873    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:13:45.703793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.701358    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 12:13:45.792724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.792536    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:13:45.925186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:45.925132    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Nov 03 12:13:46.201725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.201504    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.254930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.254869    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.274398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.267808    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:46.274398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.267891    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.382887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.382795    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:13:46.432013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.431978    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 12:13:46.441026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.440991    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" event={"ID":"d8cb4163-c911-4f2b-97f8-4643635d95c7","Type":"ContainerStarted","Data":"d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a"}
Nov 03 12:13:46.454784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.453029    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:13:46.580723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.564919    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.580723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.568019    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:13:46.618116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.618086    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:13:46.662742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.661926    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:13:46.761860 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.760161    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:13:46.840545 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:46.840480    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podStartSLOduration=6.840464686 podStartE2EDuration="6.840464686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:46.712152125 +0000 UTC m=+1375.724618816" watchObservedRunningTime="2026-11-03 12:13:46.840464686 +0000 UTC m=+1375.852931350"
Nov 03 12:13:47.375171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:47.375138    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:47.381832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:47.381555    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:47.447854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:47.447822    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:47.448911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:47.448891    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:13:53.711203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:53.711162    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"]
Nov 03 12:13:58.712828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:58.712767    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m" podUID="c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" containerName="payload-processing" containerID="cri-o://a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5" gracePeriod=25
Nov 03 12:13:59.000564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.000530    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:13:59.101693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.101662    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-routing-overlay-volume\") pod \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") "
Nov 03 12:13:59.101865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.101826    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-kube-api-access-xz5bf\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-kube-api-access-xz5bf\") pod \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") "
Nov 03 12:13:59.101940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.101885    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-plugins-config-volume\") pod \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") "
Nov 03 12:13:59.102003 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.101954    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-provider-credentials\") pod \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\" (UID: \"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55\") "
Nov 03 12:13:59.102003 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.101983    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-routing-overlay-volume" pod "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" (UID: "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:59.102260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.102232    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-plugins-config-volume" pod "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" (UID: "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:59.102328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.102312    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.104020 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.103996    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-provider-credentials" pod "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" (UID: "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:59.104089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.104023    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-kube-api-access-xz5bf" pod "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" (UID: "c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55"). InnerVolumeSpecName "kube-api-access-xz5bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:59.203524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.203477    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz5bf\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-kube-api-access-xz5bf\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.203524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.203519    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.203785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.203535    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.648638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.648603    4909 generic.go:356] "Generic (PLEG): container finished" podID="c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" containerID="a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5" exitCode=0
Nov 03 12:13:59.648869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.648656    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"
Nov 03 12:13:59.648869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.648684    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m" event={"ID":"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55","Type":"ContainerDied","Data":"a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5"}
Nov 03 12:13:59.648869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.648768    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m" event={"ID":"c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55","Type":"ContainerDied","Data":"4093eb0a7d40c52fc17936a80fcd34133e129edeefcd8056098f531ca8b1fa75"}
Nov 03 12:13:59.648869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.648790    4909 scope.go:122] "RemoveContainer" containerID="a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5"
Nov 03 12:13:59.658526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.658506    4909 scope.go:122] "RemoveContainer" containerID="a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5"
Nov 03 12:13:59.658795 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:13:59.658776    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5\": container with ID starting with a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5 not found: ID does not exist" containerID="a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5"
Nov 03 12:13:59.658869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.658814    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5"} err="failed to get container status \"a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5\": rpc error: code = NotFound desc = could not find container \"a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5\": container with ID starting with a3fdf41ac8aa29231397c1110bc0f77812cdc6224091eefaab47772c6cb47dc5 not found: ID does not exist"
Nov 03 12:13:59.900238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.900164    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"]
Nov 03 12:13:59.906400 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:13:59.906363    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bf9fbb45-tsq2m"]
Nov 03 12:14:00.269469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.269433    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"]
Nov 03 12:14:00.270603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.270577    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" containerName="payload-processing"
Nov 03 12:14:00.270782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.270767    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" containerName="payload-processing"
Nov 03 12:14:00.271038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.271025    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" containerName="payload-processing"
Nov 03 12:14:00.291328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.291283    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"]
Nov 03 12:14:00.291694 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.291664    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.340584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.340557    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-86d7ad\""
Nov 03 12:14:00.340812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.340788    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-86d7ad-dockercfg-xgkkj\""
Nov 03 12:14:00.376036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.376005    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvth\" (UniqueName: \"kubernetes.io/projected/999ebb5e-bb1c-4096-8be7-7308676cd30c-kube-api-access-bnvth\") pod \"payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.376247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.376228    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999ebb5e-bb1c-4096-8be7-7308676cd30c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.477637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.477608    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999ebb5e-bb1c-4096-8be7-7308676cd30c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.477830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.477730    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvth\" (UniqueName: \"kubernetes.io/projected/999ebb5e-bb1c-4096-8be7-7308676cd30c-kube-api-access-bnvth\") pod \"payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.478404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.478383    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999ebb5e-bb1c-4096-8be7-7308676cd30c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.530260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.530193    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvth\" (UniqueName: \"kubernetes.io/projected/999ebb5e-bb1c-4096-8be7-7308676cd30c-kube-api-access-bnvth\") pod \"payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.619465 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.619434    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:00.983876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:00.983849    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"]
Nov 03 12:14:00.984901 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:14:00.984870    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999ebb5e_bb1c_4096_8be7_7308676cd30c.slice/crio-cf5d701363d06288a78c090ff5858d27eebe39f3bcab799ba38d1c4edc81beba WatchSource:0}: Error finding container cf5d701363d06288a78c090ff5858d27eebe39f3bcab799ba38d1c4edc81beba: Status 404 returned error can't find the container with id cf5d701363d06288a78c090ff5858d27eebe39f3bcab799ba38d1c4edc81beba
Nov 03 12:14:01.124401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:01.124374    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55" path="/var/lib/kubelet/pods/c2bf6390-7841-4db0-a5a3-3bbcf6ceeb55/volumes"
Nov 03 12:14:01.666496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:01.666447    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5" event={"ID":"999ebb5e-bb1c-4096-8be7-7308676cd30c","Type":"ContainerStarted","Data":"b4c1954230b0f9504945c13efc3f4102b3c6e6a880fe6fd5000dd5bff6732cc8"}
Nov 03 12:14:01.666907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:01.666867    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5" event={"ID":"999ebb5e-bb1c-4096-8be7-7308676cd30c","Type":"ContainerStarted","Data":"cf5d701363d06288a78c090ff5858d27eebe39f3bcab799ba38d1c4edc81beba"}
Nov 03 12:14:01.667622 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:01.666932    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:01.771902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:01.771855    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5" podStartSLOduration=1.771842272 podStartE2EDuration="1.771842272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:14:01.770879839 +0000 UTC m=+1390.783346503" watchObservedRunningTime="2026-11-03 12:14:01.771842272 +0000 UTC m=+1390.784308951"
Nov 03 12:14:03.675536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:03.675501    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:14:06.222877 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.222818    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.287234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.287184    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.391724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.388053    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:14:06.544021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.543919    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 03 12:14:06.694859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.694810    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.716790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.716747    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.717493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.717028    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:14:06.740926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:06.740216    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.106841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.106791    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.232730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.232587    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.233664 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.233630    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:14:07.259711 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.259665    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:14:07.291925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.291892    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.292453 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.292427    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:14:07.606838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.606794    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.609687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.609660    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:14:07.645384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.645339    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.831722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.829008    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.831722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.829712    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:14:07.864788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.839911    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.864788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.840276    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.864788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.840583    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:14:07.864788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:07.840636    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:14:08.199327 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:08.199301    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:14:08.245798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:08.245770    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:14:08.247194 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:08.247173    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 12:14:08.271140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:08.271119    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:14:08.330725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:08.330679    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 12:14:09.566157 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:14:09.566122    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:14:12.675575 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:12.675545    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:14:25.170550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.170517    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"]
Nov 03 12:14:25.175031 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.175009    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.270246 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.270210    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ac6071-serving-cert\""
Nov 03 12:14:25.274618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.274590    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"]
Nov 03 12:14:25.312299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.312257    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c141b778-28d5-44b8-bc3c-5b762195ffb6-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.312438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.312404    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-metrics-certs\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.312520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.312504    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-maas-api-tls\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.312592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.312576    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ntx\" (UniqueName: \"kubernetes.io/projected/c141b778-28d5-44b8-bc3c-5b762195ffb6-kube-api-access-g7ntx\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.417309 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.417254    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c141b778-28d5-44b8-bc3c-5b762195ffb6-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.432980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.430401    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-metrics-certs\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.432980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.430518    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-maas-api-tls\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.432980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.430582    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ntx\" (UniqueName: \"kubernetes.io/projected/c141b778-28d5-44b8-bc3c-5b762195ffb6-kube-api-access-g7ntx\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.432980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.418782    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c141b778-28d5-44b8-bc3c-5b762195ffb6-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.451570 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.451533    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-maas-api-tls\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.452740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.452664    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-metrics-certs\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.495014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.494979    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ntx\" (UniqueName: \"kubernetes.io/projected/c141b778-28d5-44b8-bc3c-5b762195ffb6-kube-api-access-g7ntx\") pod \"maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:25.787928 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:25.787896    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:26.157385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:26.157356    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"]
Nov 03 12:14:26.159951 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:14:26.159926    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc141b778_28d5_44b8_bc3c_5b762195ffb6.slice/crio-ad9f3b61b95aadaf76e14cb46ec4d60acb1a948e3a34f7d1094611fb0e683a69 WatchSource:0}: Error finding container ad9f3b61b95aadaf76e14cb46ec4d60acb1a948e3a34f7d1094611fb0e683a69: Status 404 returned error can't find the container with id ad9f3b61b95aadaf76e14cb46ec4d60acb1a948e3a34f7d1094611fb0e683a69
Nov 03 12:14:26.161217 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:26.161198    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:14:27.182431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:27.182390    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9" event={"ID":"c141b778-28d5-44b8-bc3c-5b762195ffb6","Type":"ContainerStarted","Data":"a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313"}
Nov 03 12:14:27.183209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:27.183176    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9" event={"ID":"c141b778-28d5-44b8-bc3c-5b762195ffb6","Type":"ContainerStarted","Data":"ad9f3b61b95aadaf76e14cb46ec4d60acb1a948e3a34f7d1094611fb0e683a69"}
Nov 03 12:14:27.183362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:27.183327    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:27.293614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:27.293523    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9" podStartSLOduration=1.497166384 podStartE2EDuration="2.29351014s" totalImagesPullingTime="796.343756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:14:27.290339951 +0000 UTC m=+1416.302806613" watchObservedRunningTime="2026-11-03 12:14:27.29351014 +0000 UTC m=+1416.305976803"
Nov 03 12:14:29.656079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.656047    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"]
Nov 03 12:14:29.660393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.660369    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.755651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.755620    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-86d7ad-serving-cert\""
Nov 03 12:14:29.758931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.758905    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"]
Nov 03 12:14:29.816888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.816857    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-maas-api-tls\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.816888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.816896    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-openshift-service-ca\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.817130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.817110    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-metrics-certs\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.817178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.817161    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrqt\" (UniqueName: \"kubernetes.io/projected/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-kube-api-access-lzrqt\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.939248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.939208    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-maas-api-tls\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.942686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.939484    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-openshift-service-ca\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.954722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.943181    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-metrics-certs\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.955038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.955011    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrqt\" (UniqueName: \"kubernetes.io/projected/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-kube-api-access-lzrqt\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.957782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.948116    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-openshift-service-ca\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.996770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.978295    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-metrics-certs\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:29.996770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:29.978341    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-maas-api-tls\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:30.009821 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:30.009786    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrqt\" (UniqueName: \"kubernetes.io/projected/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-kube-api-access-lzrqt\") pod \"maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:30.274539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:30.274466    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:30.642091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:30.642067    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"]
Nov 03 12:14:30.643435 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:14:30.643404    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b7517f_8ab0_45f7_990b_f2636ddb5fe2.slice/crio-28e192511173e0d729330eb2a4502dc250224090a563b31bf09614c4a7afa1ef WatchSource:0}: Error finding container 28e192511173e0d729330eb2a4502dc250224090a563b31bf09614c4a7afa1ef: Status 404 returned error can't find the container with id 28e192511173e0d729330eb2a4502dc250224090a563b31bf09614c4a7afa1ef
Nov 03 12:14:31.207792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:31.207761    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7" event={"ID":"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2","Type":"ContainerStarted","Data":"28e192511173e0d729330eb2a4502dc250224090a563b31bf09614c4a7afa1ef"}
Nov 03 12:14:32.216157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:32.216120    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7" event={"ID":"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2","Type":"ContainerStarted","Data":"ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d"}
Nov 03 12:14:32.216532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:32.216287    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:32.324206 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:32.324155    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7" podStartSLOduration=2.475380211 podStartE2EDuration="3.324141986s" totalImagesPullingTime="848.761775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:14:32.321640069 +0000 UTC m=+1421.334106733" watchObservedRunningTime="2026-11-03 12:14:32.324141986 +0000 UTC m=+1421.336608678"
Nov 03 12:14:33.195021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:33.194993    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:14:38.227931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:38.227899    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:14:39.564098 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:14:39.564060    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:14:42.526898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:14:42.526836    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:15:00.182541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.182506    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"]
Nov 03 12:15:00.188524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.188501    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.237269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.237240    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:15:00.288485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.288454    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"]
Nov 03 12:15:00.332478 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.332445    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e411da27-828d-430a-befa-3dc432ff90ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.332633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.332620    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8d7\" (UniqueName: \"kubernetes.io/projected/e411da27-828d-430a-befa-3dc432ff90ee-kube-api-access-6n8d7\") pod \"maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.434398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.434317    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e411da27-828d-430a-befa-3dc432ff90ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.434551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.434454    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8d7\" (UniqueName: \"kubernetes.io/projected/e411da27-828d-430a-befa-3dc432ff90ee-kube-api-access-6n8d7\") pod \"maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.435164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.435138    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e411da27-828d-430a-befa-3dc432ff90ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.487019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.486995    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8d7\" (UniqueName: \"kubernetes.io/projected/e411da27-828d-430a-befa-3dc432ff90ee-kube-api-access-6n8d7\") pod \"maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.502978 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.502958    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:00.868613 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:00.868586    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"]
Nov 03 12:15:00.870278 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:15:00.870251    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode411da27_828d_430a_befa_3dc432ff90ee.slice/crio-9cc4d4f784084456d81a766fa31d4355b9227b8c42bf6961d1d4968efc695c45 WatchSource:0}: Error finding container 9cc4d4f784084456d81a766fa31d4355b9227b8c42bf6961d1d4968efc695c45: Status 404 returned error can't find the container with id 9cc4d4f784084456d81a766fa31d4355b9227b8c42bf6961d1d4968efc695c45
Nov 03 12:15:01.448905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:01.448867    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" event={"ID":"e411da27-828d-430a-befa-3dc432ff90ee","Type":"ContainerStarted","Data":"9cc4d4f784084456d81a766fa31d4355b9227b8c42bf6961d1d4968efc695c45"}
Nov 03 12:15:04.318730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:04.318669    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:15:07.499004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:07.498971    4909 generic.go:356] "Generic (PLEG): container finished" podID="e411da27-828d-430a-befa-3dc432ff90ee" containerID="f937eb55b23708e7eeb88e21c4a08611eb29c4838b487a29e609ad94cc7fb300" exitCode=60
Nov 03 12:15:07.499357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:07.499058    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" event={"ID":"e411da27-828d-430a-befa-3dc432ff90ee","Type":"ContainerDied","Data":"f937eb55b23708e7eeb88e21c4a08611eb29c4838b487a29e609ad94cc7fb300"}
Nov 03 12:15:07.499357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:07.499309    4909 scope.go:122] "RemoveContainer" containerID="f937eb55b23708e7eeb88e21c4a08611eb29c4838b487a29e609ad94cc7fb300"
Nov 03 12:15:08.507935 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:08.507898    4909 generic.go:356] "Generic (PLEG): container finished" podID="e411da27-828d-430a-befa-3dc432ff90ee" containerID="4e7dc927eef77cca0d2578d7c6b0ef6efcae13d20c0178358f0545d6e242a606" exitCode=60
Nov 03 12:15:08.508397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:08.507943    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" event={"ID":"e411da27-828d-430a-befa-3dc432ff90ee","Type":"ContainerDied","Data":"4e7dc927eef77cca0d2578d7c6b0ef6efcae13d20c0178358f0545d6e242a606"}
Nov 03 12:15:08.508397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:08.508031    4909 scope.go:122] "RemoveContainer" containerID="f937eb55b23708e7eeb88e21c4a08611eb29c4838b487a29e609ad94cc7fb300"
Nov 03 12:15:08.508397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:08.508301    4909 scope.go:122] "RemoveContainer" containerID="4e7dc927eef77cca0d2578d7c6b0ef6efcae13d20c0178358f0545d6e242a606"
Nov 03 12:15:08.508680 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:15:08.508654    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct_odh-ai-gateway-infra(e411da27-828d-430a-befa-3dc432ff90ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" podUID="e411da27-828d-430a-befa-3dc432ff90ee"
Nov 03 12:15:09.516271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:09.516240    4909 scope.go:122] "RemoveContainer" containerID="4e7dc927eef77cca0d2578d7c6b0ef6efcae13d20c0178358f0545d6e242a606"
Nov 03 12:15:09.516688 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:15:09.516558    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct_odh-ai-gateway-infra(e411da27-828d-430a-befa-3dc432ff90ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" podUID="e411da27-828d-430a-befa-3dc432ff90ee"
Nov 03 12:15:09.584601 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:15:09.584566    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:15:18.373756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.372161    4909 patch_prober.go:29] interesting pod/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:57282->10.132.0.81:15021: read: connection reset by peer" start-of-body=
Nov 03 12:15:18.373756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.372376    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:57282->10.132.0.81:15021: read: connection reset by peer"
Nov 03 12:15:18.594006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.593918    4909 generic.go:356] "Generic (PLEG): container finished" podID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerID="d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a" exitCode=0
Nov 03 12:15:18.594006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.593962    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" event={"ID":"d8cb4163-c911-4f2b-97f8-4643635d95c7","Type":"ContainerDied","Data":"d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a"}
Nov 03 12:15:18.594396 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.594379    4909 scope.go:122] "RemoveContainer" containerID="d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a"
Nov 03 12:15:18.595943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.595917    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:18.596001 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.595983    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:18.596037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:18.596016    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:19.835525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:19.835483    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" event={"ID":"d8cb4163-c911-4f2b-97f8-4643635d95c7","Type":"ContainerStarted","Data":"db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05"}
Nov 03 12:15:20.375678 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:20.375637    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:15:20.376425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:20.376403    4909 patch_prober.go:29] interesting pod/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:15:20.376481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:20.376463    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 12:15:21.375386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:21.375350    4909 patch_prober.go:29] interesting pod/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:15:21.375800 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:21.375407    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 12:15:22.099972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.099932    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"]
Nov 03 12:15:22.105952 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.105931    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.115985 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.115964    4909 scope.go:122] "RemoveContainer" containerID="4e7dc927eef77cca0d2578d7c6b0ef6efcae13d20c0178358f0545d6e242a606"
Nov 03 12:15:22.170958 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.170925    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-certs\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.171214 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.171194    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istiod-ca-cert\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.171386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.171368    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-credential-socket\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.176617 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.176591    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-socket\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.176768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.176627    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-envoy\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.176768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.176724    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-podinfo\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.176768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.176751    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-token\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.176951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.176773    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-data\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.176951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.176808    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrxd\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-kube-api-access-znrxd\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.176951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.176211    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-931523-openshift-default-dockercfg-89jzf\""
Nov 03 12:15:22.214962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.214937    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"]
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.291669    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-socket\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.291769    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-envoy\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.291857    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-podinfo\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.291894    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-token\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.291931    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-data\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.291984    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrxd\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-kube-api-access-znrxd\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.292194    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-certs\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.292238    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istiod-ca-cert\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.294260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.292318    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-credential-socket\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.333117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.333082    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-credential-socket\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.389775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.355593    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istiod-ca-cert\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.389775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.361955    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-socket\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.389775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.362137    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-certs\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.389775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.380298    4909 patch_prober.go:29] interesting pod/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:15:22.389775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.380359    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 12:15:22.403250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.403189    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-envoy\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.423775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.411022    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-data\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.423775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.422331    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-podinfo\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.495721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.495331    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-token\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.573071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.570985    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrxd\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-kube-api-access-znrxd\") pod \"e2e-api-b-931523-openshift-default-6b754b545c-czpt9\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") " pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.725741 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.723203    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:22.880326 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.880293    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" event={"ID":"e411da27-828d-430a-befa-3dc432ff90ee","Type":"ContainerStarted","Data":"beda153987a7b559c54a0e0a52f1bc31a7cb887b646f399599ec21e332903393"}
Nov 03 12:15:22.985966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:22.985867    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" podStartSLOduration=17.321791512 podStartE2EDuration="22.985852002s" totalImagesPullingTime="5.66406049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:22.984521335 +0000 UTC m=+1471.996988012" watchObservedRunningTime="2026-11-03 12:15:22.985852002 +0000 UTC m=+1471.998318665"
Nov 03 12:15:23.182201 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.182169    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"]
Nov 03 12:15:23.183339 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:15:23.183296    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e18d399_8a08_4e65_9310_b2cb6c7d1d63.slice/crio-84be67745e9315113f01b26b34457626c69519ddd9255d2fdfb68ba2d30da1f9 WatchSource:0}: Error finding container 84be67745e9315113f01b26b34457626c69519ddd9255d2fdfb68ba2d30da1f9: Status 404 returned error can't find the container with id 84be67745e9315113f01b26b34457626c69519ddd9255d2fdfb68ba2d30da1f9
Nov 03 12:15:23.186199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.186162    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:23.186310 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.186242    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:23.186310 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.186288    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:23.377237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.377109    4909 patch_prober.go:29] interesting pod/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:15:23.377237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.377164    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 12:15:23.567773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.566858    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"]
Nov 03 12:15:23.572345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.572323    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.612337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.612302    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-home\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.612946 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.612912    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-tmp-dir\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.613256 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.613238    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9p8\" (UniqueName: \"kubernetes.io/projected/955970d0-cd2b-46b5-8d24-4cf06cf33989-kube-api-access-rf9p8\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.613553 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.613537    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955970d0-cd2b-46b5-8d24-4cf06cf33989-tls-certs\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.613754 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.613739    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-model-cache\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.614103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.614070    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-dshm\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.621181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.620851    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"distinct-2-w2-91cd99-kserve-self-signed-certs\""
Nov 03 12:15:23.621446 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.621427    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"]
Nov 03 12:15:23.724718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.723968    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-dshm\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.725124    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-home\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.725191    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-home\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.725222    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-tmp-dir\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.725277    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9p8\" (UniqueName: \"kubernetes.io/projected/955970d0-cd2b-46b5-8d24-4cf06cf33989-kube-api-access-rf9p8\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.725970    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-tmp-dir\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.727040    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955970d0-cd2b-46b5-8d24-4cf06cf33989-tls-certs\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.727143    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-model-cache\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.733337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.731984    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-model-cache\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.780718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.780081    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955970d0-cd2b-46b5-8d24-4cf06cf33989-tls-certs\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.801830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.791427    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9p8\" (UniqueName: \"kubernetes.io/projected/955970d0-cd2b-46b5-8d24-4cf06cf33989-kube-api-access-rf9p8\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.820799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.803573    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-dshm\") pod \"distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.885489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.885073    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:23.891595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.891571    4909 generic.go:356] "Generic (PLEG): container finished" podID="e411da27-828d-430a-befa-3dc432ff90ee" containerID="beda153987a7b559c54a0e0a52f1bc31a7cb887b646f399599ec21e332903393" exitCode=60
Nov 03 12:15:23.891684 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.891631    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" event={"ID":"e411da27-828d-430a-befa-3dc432ff90ee","Type":"ContainerDied","Data":"beda153987a7b559c54a0e0a52f1bc31a7cb887b646f399599ec21e332903393"}
Nov 03 12:15:23.891684 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.891672    4909 scope.go:122] "RemoveContainer" containerID="4e7dc927eef77cca0d2578d7c6b0ef6efcae13d20c0178358f0545d6e242a606"
Nov 03 12:15:23.892091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.892077    4909 scope.go:122] "RemoveContainer" containerID="beda153987a7b559c54a0e0a52f1bc31a7cb887b646f399599ec21e332903393"
Nov 03 12:15:23.892354 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:15:23.892338    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct_odh-ai-gateway-infra(e411da27-828d-430a-befa-3dc432ff90ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" podUID="e411da27-828d-430a-befa-3dc432ff90ee"
Nov 03 12:15:23.894639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.894617    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" event={"ID":"3e18d399-8a08-4e65-9310-b2cb6c7d1d63","Type":"ContainerStarted","Data":"62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597"}
Nov 03 12:15:23.894736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:23.894657    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" event={"ID":"3e18d399-8a08-4e65-9310-b2cb6c7d1d63","Type":"ContainerStarted","Data":"84be67745e9315113f01b26b34457626c69519ddd9255d2fdfb68ba2d30da1f9"}
Nov 03 12:15:24.040722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.037800    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"]
Nov 03 12:15:24.412560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.412473    4909 patch_prober.go:29] interesting pod/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:15:24.412560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.412540    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 12:15:24.464415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.464341    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podStartSLOduration=2.464325155 podStartE2EDuration="2.464325155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:24.172214571 +0000 UTC m=+1473.184681238" watchObservedRunningTime="2026-11-03 12:15:24.464325155 +0000 UTC m=+1473.476791823"
Nov 03 12:15:24.469350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.469323    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"]
Nov 03 12:15:24.471421 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:15:24.470836    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod955970d0_cd2b_46b5_8d24_4cf06cf33989.slice/crio-d16ebebc38e5f6f3674f9ccd46f3dc1fd34e04aa978b9a3a1db64cd1e01d1619 WatchSource:0}: Error finding container d16ebebc38e5f6f3674f9ccd46f3dc1fd34e04aa978b9a3a1db64cd1e01d1619: Status 404 returned error can't find the container with id d16ebebc38e5f6f3674f9ccd46f3dc1fd34e04aa978b9a3a1db64cd1e01d1619
Nov 03 12:15:24.733321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.731784    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:24.761794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.742496    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:24.974809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.974759    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" event={"ID":"955970d0-cd2b-46b5-8d24-4cf06cf33989","Type":"ContainerStarted","Data":"7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea"}
Nov 03 12:15:24.974957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.974841    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" event={"ID":"955970d0-cd2b-46b5-8d24-4cf06cf33989","Type":"ContainerStarted","Data":"d16ebebc38e5f6f3674f9ccd46f3dc1fd34e04aa978b9a3a1db64cd1e01d1619"}
Nov 03 12:15:24.974957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.974886    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:24.978639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:24.978029    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:15:25.166552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.166527    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:25.276947 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.276917    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e411da27-828d-430a-befa-3dc432ff90ee-kube-api-access-6n8d7\" (UniqueName: \"kubernetes.io/projected/e411da27-828d-430a-befa-3dc432ff90ee-kube-api-access-6n8d7\") pod \"e411da27-828d-430a-befa-3dc432ff90ee\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") "
Nov 03 12:15:25.277176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.277013    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e411da27-828d-430a-befa-3dc432ff90ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e411da27-828d-430a-befa-3dc432ff90ee-maas-api-service-ca\") pod \"e411da27-828d-430a-befa-3dc432ff90ee\" (UID: \"e411da27-828d-430a-befa-3dc432ff90ee\") "
Nov 03 12:15:25.277710 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.277664    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e411da27-828d-430a-befa-3dc432ff90ee-maas-api-service-ca" pod "e411da27-828d-430a-befa-3dc432ff90ee" (UID: "e411da27-828d-430a-befa-3dc432ff90ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:25.279143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.279120    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e411da27-828d-430a-befa-3dc432ff90ee-kube-api-access-6n8d7" pod "e411da27-828d-430a-befa-3dc432ff90ee" (UID: "e411da27-828d-430a-befa-3dc432ff90ee"). InnerVolumeSpecName "kube-api-access-6n8d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:25.288844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.288765    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" podStartSLOduration=2.288752425 podStartE2EDuration="2.288752425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:25.286401536 +0000 UTC m=+1474.298868213" watchObservedRunningTime="2026-11-03 12:15:25.288752425 +0000 UTC m=+1474.301219088"
Nov 03 12:15:25.378743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.378694    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8d7\" (UniqueName: \"kubernetes.io/projected/e411da27-828d-430a-befa-3dc432ff90ee-kube-api-access-6n8d7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:25.378743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.378738    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e411da27-828d-430a-befa-3dc432ff90ee-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:25.597793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.597676    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:15:25.598474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.598446    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:15:25.599394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.599343    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:15:25.885991 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.885902    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:25.981566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.981537    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"
Nov 03 12:15:25.981743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.981601    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct" event={"ID":"e411da27-828d-430a-befa-3dc432ff90ee","Type":"ContainerDied","Data":"9cc4d4f784084456d81a766fa31d4355b9227b8c42bf6961d1d4968efc695c45"}
Nov 03 12:15:25.981743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:25.981659    4909 scope.go:122] "RemoveContainer" containerID="beda153987a7b559c54a0e0a52f1bc31a7cb887b646f399599ec21e332903393"
Nov 03 12:15:26.133831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:26.133794    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"]
Nov 03 12:15:26.136999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:26.136949    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-86d7ad-29895135-2mmct"]
Nov 03 12:15:27.123827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:27.123800    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e411da27-828d-430a-befa-3dc432ff90ee" path="/var/lib/kubelet/pods/e411da27-828d-430a-befa-3dc432ff90ee/volumes"
Nov 03 12:15:33.646127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:33.646048    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:15:33.885325 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:33.885280    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:33.898563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:33.898497    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:34.127969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:34.127935    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 12:15:38.888061 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888031    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"]
Nov 03 12:15:38.888818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888798    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.888818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888818    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.888938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888848    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.888938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888856    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.888938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888865    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.888938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888870    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.889068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.888994    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.889068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.889006    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.889068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.889011    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e411da27-828d-430a-befa-3dc432ff90ee" containerName="cleanup"
Nov 03 12:15:38.895871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.895846    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:38.992093 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.992059    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f6c034-openshift-default-dockercfg-2dcdl\""
Nov 03 12:15:38.995373 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:38.995344    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"]
Nov 03 12:15:39.034966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.034924    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-socket\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035055    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-data\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035121    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-credential-socket\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035171    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-token\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035232    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-certs\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035309 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035271    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-envoy\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035309 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035291    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-podinfo\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035386    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995db966-94b9-4691-a0ec-c5e88b54d7bb-istiod-ca-cert\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.035426 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.035403    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd27w\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-kube-api-access-kd27w\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.151209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.151127    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-certs\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.175058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.174916    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-envoy\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.183600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.183568    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-podinfo\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.184236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.184216    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995db966-94b9-4691-a0ec-c5e88b54d7bb-istiod-ca-cert\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.184578 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.184556    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd27w\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-kube-api-access-kd27w\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.185166 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.185083    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-socket\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.185457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.185437    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-data\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.185659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.185644    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-credential-socket\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.185805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.185788    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-token\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.186314 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.174741    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-certs\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.193340 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.193314    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-socket\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.212725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.195166    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995db966-94b9-4691-a0ec-c5e88b54d7bb-istiod-ca-cert\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.212725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.195784    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-data\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.212725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.196030    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-credential-socket\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.213962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.213932    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-podinfo\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.224903 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.224874    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-envoy\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.256593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.256554    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-token\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.278563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.270523    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd27w\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-kube-api-access-kd27w\") pod \"e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") " pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.510754 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.510720    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:39.572669 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:15:39.572634    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:15:39.928346 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.928275    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"]
Nov 03 12:15:39.966797 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:15:39.966770    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995db966_94b9_4691_a0ec_c5e88b54d7bb.slice/crio-8a80ccdf5eca47e0321ca161094f590cb3d9f623dc2446cd4c8bb121a18c8989 WatchSource:0}: Error finding container 8a80ccdf5eca47e0321ca161094f590cb3d9f623dc2446cd4c8bb121a18c8989: Status 404 returned error can't find the container with id 8a80ccdf5eca47e0321ca161094f590cb3d9f623dc2446cd4c8bb121a18c8989
Nov 03 12:15:39.969416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.969384    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:39.969508 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.969452    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:39.969508 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:39.969484    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:40.184760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:40.184658    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" event={"ID":"995db966-94b9-4691-a0ec-c5e88b54d7bb","Type":"ContainerStarted","Data":"04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b"}
Nov 03 12:15:40.184760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:40.184739    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" event={"ID":"995db966-94b9-4691-a0ec-c5e88b54d7bb","Type":"ContainerStarted","Data":"8a80ccdf5eca47e0321ca161094f590cb3d9f623dc2446cd4c8bb121a18c8989"}
Nov 03 12:15:40.338513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:40.338440    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podStartSLOduration=2.338421626 podStartE2EDuration="2.338421626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:40.291145715 +0000 UTC m=+1489.303612378" watchObservedRunningTime="2026-11-03 12:15:40.338421626 +0000 UTC m=+1489.350888290"
Nov 03 12:15:40.513099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:40.511324    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:41.479741 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.479692    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"]
Nov 03 12:15:41.483897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.483878    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.517476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.517453    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:41.521093 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521071    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521130    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521265    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521302    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521344    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qr95\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-kube-api-access-9qr95\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521400 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521387    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9044e50e-4ae3-4735-a37e-408156f3b9e1-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521480    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521583    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.521638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.521626    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.532001 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.531981    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-69fd8043-openshift-default-dockercfg-qfwfh\""
Nov 03 12:15:41.581897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.581869    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"]
Nov 03 12:15:41.622767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.622737    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.622889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.622830    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qr95\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-kube-api-access-9qr95\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.622889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.622864    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9044e50e-4ae3-4735-a37e-408156f3b9e1-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.623016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.622923    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.623066 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.623038    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.623119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.623104    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.623207 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.623182    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.623271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.623244    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.623320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.623288    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.623363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.623341    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.624028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.624004    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.624108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.624089    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.624834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.624782    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9044e50e-4ae3-4735-a37e-408156f3b9e1-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.625053 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.625030    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.626558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.626529    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.626655 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.626573    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.676329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.676307    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.676500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.676482    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qr95\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-kube-api-access-9qr95\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:41.798176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:41.798107    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:42.197494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:42.197460    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:42.199158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:42.199132    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:15:42.215660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:42.215637    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"]
Nov 03 12:15:42.218873 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:15:42.218840    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9044e50e_4ae3_4735_a37e_408156f3b9e1.slice/crio-a1476c2cb34dbb9945337da5006fd21841d4d31be49d7a2d740c1f2a184a3e4c WatchSource:0}: Error finding container a1476c2cb34dbb9945337da5006fd21841d4d31be49d7a2d740c1f2a184a3e4c: Status 404 returned error can't find the container with id a1476c2cb34dbb9945337da5006fd21841d4d31be49d7a2d740c1f2a184a3e4c
Nov 03 12:15:42.221159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:42.221127    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:42.221264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:42.221201    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:42.221264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:42.221247    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:15:43.205798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:43.205758    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" event={"ID":"9044e50e-4ae3-4735-a37e-408156f3b9e1","Type":"ContainerStarted","Data":"192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2"}
Nov 03 12:15:43.206224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:43.205826    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" event={"ID":"9044e50e-4ae3-4735-a37e-408156f3b9e1","Type":"ContainerStarted","Data":"a1476c2cb34dbb9945337da5006fd21841d4d31be49d7a2d740c1f2a184a3e4c"}
Nov 03 12:15:43.332797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:43.328880    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podStartSLOduration=2.328856665 podStartE2EDuration="2.328856665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:43.322014716 +0000 UTC m=+1492.334481379" watchObservedRunningTime="2026-11-03 12:15:43.328856665 +0000 UTC m=+1492.341323332"
Nov 03 12:15:43.798621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:43.798591    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:43.800110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:43.800085    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 12:15:43.800233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:43.800133    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 12:15:44.798637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:44.798594    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 12:15:44.798637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:44.798637    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 12:15:45.799046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:45.799010    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 12:15:45.799437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:45.799069    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 12:15:46.990876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:46.990835    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 12:15:46.992438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:46.991458    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 12:15:47.798582 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:47.798547    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 12:15:47.798782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:47.798600    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 12:15:48.798964 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:48.798926    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 12:15:48.799375 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:48.798980    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 12:15:49.802445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:49.802413    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:49.802934 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:49.802820    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:49.803651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:49.803629    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:15:51.303779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:51.303751    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:15:51.304688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:15:51.304669    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:16:09.564332 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:16:09.564294    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:16:30.442723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:30.426016    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:16:32.820542 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:32.820507    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"]
Nov 03 12:16:32.918317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:32.918284    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"]
Nov 03 12:16:37.821391 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:37.821282    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" podUID="4e20ef1f-cf79-4c2d-9374-90106a25e594" containerName="payload-processing" containerID="cri-o://8fbab4255081e28ee679662edfd33f05e54b821c953437d230a4a455e5da52c3" gracePeriod=25
Nov 03 12:16:37.919122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:37.919069    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" podUID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" containerName="payload-pre-processing" containerID="cri-o://1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b" gracePeriod=25
Nov 03 12:16:39.565769 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:16:39.565730    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:16:41.007749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.007265    4909 patch_prober.go:29] interesting pod/e2e-api-b-931523-openshift-default-6b754b545c-czpt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:16:41.007749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.007362    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:16:41.862360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.862328    4909 generic.go:356] "Generic (PLEG): container finished" podID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerID="62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597" exitCode=0
Nov 03 12:16:41.862544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.862400    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" event={"ID":"3e18d399-8a08-4e65-9310-b2cb6c7d1d63","Type":"ContainerDied","Data":"62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597"}
Nov 03 12:16:41.862779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.862764    4909 scope.go:122] "RemoveContainer" containerID="62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597"
Nov 03 12:16:41.864175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.864146    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:16:41.864300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.864226    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:16:41.864300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:41.864273    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:16:42.870933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:42.870856    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" event={"ID":"3e18d399-8a08-4e65-9310-b2cb6c7d1d63","Type":"ContainerStarted","Data":"d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57"}
Nov 03 12:16:43.726300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:43.726264    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:16:43.726509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:43.726432    4909 patch_prober.go:29] interesting pod/e2e-api-b-931523-openshift-default-6b754b545c-czpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:16:43.726509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:43.726482    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 12:16:44.726478 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:44.726445    4909 patch_prober.go:29] interesting pod/e2e-api-b-931523-openshift-default-6b754b545c-czpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:16:44.726912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:44.726493    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 12:16:45.534762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:45.534727    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" podUID="4e20ef1f-cf79-4c2d-9374-90106a25e594" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:16:45.726648 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:45.726614    4909 patch_prober.go:29] interesting pod/e2e-api-b-931523-openshift-default-6b754b545c-czpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:16:45.727072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:45.726664    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 12:16:46.562863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:46.562817    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" podUID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:16:46.727015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:46.726979    4909 patch_prober.go:29] interesting pod/e2e-api-b-931523-openshift-default-6b754b545c-czpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:16:46.727450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:46.727048    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 12:16:47.727734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:47.727686    4909 patch_prober.go:29] interesting pod/e2e-api-b-931523-openshift-default-6b754b545c-czpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:16:47.728306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:47.728276    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 12:16:48.730607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:48.730577    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:16:48.731089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:48.730939    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:16:48.731809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:48.731791    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:16:55.534620 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:55.534579    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" podUID="4e20ef1f-cf79-4c2d-9374-90106a25e594" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:16:56.540228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:56.540191    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" podUID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:16:57.981223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:57.981196    4909 generic.go:356] "Generic (PLEG): container finished" podID="4e20ef1f-cf79-4c2d-9374-90106a25e594" containerID="8fbab4255081e28ee679662edfd33f05e54b821c953437d230a4a455e5da52c3" exitCode=0
Nov 03 12:16:57.981549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:57.981245    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" event={"ID":"4e20ef1f-cf79-4c2d-9374-90106a25e594","Type":"ContainerDied","Data":"8fbab4255081e28ee679662edfd33f05e54b821c953437d230a4a455e5da52c3"}
Nov 03 12:16:58.193204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.192933    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:16:58.227571 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.227546    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:16:58.237228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.237208    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e20ef1f-cf79-4c2d-9374-90106a25e594-kube-api-access-zqnt4\" (UniqueName: \"kubernetes.io/projected/4e20ef1f-cf79-4c2d-9374-90106a25e594-kube-api-access-zqnt4\") pod \"4e20ef1f-cf79-4c2d-9374-90106a25e594\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") "
Nov 03 12:16:58.237318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.237306    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e20ef1f-cf79-4c2d-9374-90106a25e594-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e20ef1f-cf79-4c2d-9374-90106a25e594-plugins-config-volume\") pod \"4e20ef1f-cf79-4c2d-9374-90106a25e594\" (UID: \"4e20ef1f-cf79-4c2d-9374-90106a25e594\") "
Nov 03 12:16:58.237657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.237634    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e20ef1f-cf79-4c2d-9374-90106a25e594-plugins-config-volume" pod "4e20ef1f-cf79-4c2d-9374-90106a25e594" (UID: "4e20ef1f-cf79-4c2d-9374-90106a25e594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:16:58.238036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.238013    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e20ef1f-cf79-4c2d-9374-90106a25e594-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:58.239292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.239271    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e20ef1f-cf79-4c2d-9374-90106a25e594-kube-api-access-zqnt4" pod "4e20ef1f-cf79-4c2d-9374-90106a25e594" (UID: "4e20ef1f-cf79-4c2d-9374-90106a25e594"). InnerVolumeSpecName "kube-api-access-zqnt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:16:58.339127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.339053    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-kube-api-access-6hdrb\" (UniqueName: \"kubernetes.io/projected/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-kube-api-access-6hdrb\") pod \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") "
Nov 03 12:16:58.339430 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.339404    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-plugins-config-volume\") pod \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\" (UID: \"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3\") "
Nov 03 12:16:58.339820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.339795    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-plugins-config-volume" pod "9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" (UID: "9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:16:58.339992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.339976    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqnt4\" (UniqueName: \"kubernetes.io/projected/4e20ef1f-cf79-4c2d-9374-90106a25e594-kube-api-access-zqnt4\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:58.340089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.339994    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:58.341511 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.341491    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-kube-api-access-6hdrb" pod "9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" (UID: "9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3"). InnerVolumeSpecName "kube-api-access-6hdrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:16:58.441483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.441442    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hdrb\" (UniqueName: \"kubernetes.io/projected/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3-kube-api-access-6hdrb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:58.988857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.988824    4909 generic.go:356] "Generic (PLEG): container finished" podID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" containerID="1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b" exitCode=0
Nov 03 12:16:58.989283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.988887    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"
Nov 03 12:16:58.989283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.988906    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" event={"ID":"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3","Type":"ContainerDied","Data":"1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b"}
Nov 03 12:16:58.989283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.988994    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf" event={"ID":"9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3","Type":"ContainerDied","Data":"991814c8b4566469d2defb69798e99728bf7f88e6b0a0e49705ef58b57d5ee0d"}
Nov 03 12:16:58.989283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.989023    4909 scope.go:122] "RemoveContainer" containerID="1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b"
Nov 03 12:16:58.990977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.990844    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf" event={"ID":"4e20ef1f-cf79-4c2d-9374-90106a25e594","Type":"ContainerDied","Data":"f7cf673060a59bdb21a3c8023188445af430d04661dac247393c5ac6954d9cd6"}
Nov 03 12:16:58.990977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.990951    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"
Nov 03 12:16:58.993734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.993681    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:16:58.999965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:58.999948    4909 scope.go:122] "RemoveContainer" containerID="1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b"
Nov 03 12:16:59.000260 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:16:59.000242    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b\": container with ID starting with 1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b not found: ID does not exist" containerID="1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b"
Nov 03 12:16:59.000315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:59.000268    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b"} err="failed to get container status \"1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b\": rpc error: code = NotFound desc = could not find container \"1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b\": container with ID starting with 1033bdefc078a4ba645101923a0186407e69035104e0d067b2e07bca92c0227b not found: ID does not exist"
Nov 03 12:16:59.000315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:59.000292    4909 scope.go:122] "RemoveContainer" containerID="8fbab4255081e28ee679662edfd33f05e54b821c953437d230a4a455e5da52c3"
Nov 03 12:16:59.141267 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:59.141196    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"]
Nov 03 12:16:59.144448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:59.144426    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-lxmkf"]
Nov 03 12:16:59.244493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:59.244464    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"]
Nov 03 12:16:59.255096 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:16:59.255063    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-kz4rf"]
Nov 03 12:17:01.124653 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:01.124621    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e20ef1f-cf79-4c2d-9374-90106a25e594" path="/var/lib/kubelet/pods/4e20ef1f-cf79-4c2d-9374-90106a25e594/volumes"
Nov 03 12:17:01.125104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:01.125087    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" path="/var/lib/kubelet/pods/9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3/volumes"
Nov 03 12:17:05.684861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.684825    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"]
Nov 03 12:17:05.685725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.685677    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" containerName="payload-pre-processing"
Nov 03 12:17:05.685846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.685762    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" containerName="payload-pre-processing"
Nov 03 12:17:05.685846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.685791    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4e20ef1f-cf79-4c2d-9374-90106a25e594" containerName="payload-processing"
Nov 03 12:17:05.685846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.685800    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e20ef1f-cf79-4c2d-9374-90106a25e594" containerName="payload-processing"
Nov 03 12:17:05.686015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.685994    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1daa27-fbe3-4b6e-b32f-52fe3017e1a3" containerName="payload-pre-processing"
Nov 03 12:17:05.686015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.686014    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e20ef1f-cf79-4c2d-9374-90106a25e594" containerName="payload-processing"
Nov 03 12:17:05.691621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.690913    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.783721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.778233    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"]
Nov 03 12:17:05.839734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839338    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.839734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839414    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.839734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839450    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8hs\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-kube-api-access-vp8hs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.839734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839614    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.839734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839652    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.839734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839676    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91d447b-7696-4a75-9cdd-56dd77134e79-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.839734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839724    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.848448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839819    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.848448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.839873    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963491 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963440    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963506    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963542    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8hs\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-kube-api-access-vp8hs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963663    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963731    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963759    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91d447b-7696-4a75-9cdd-56dd77134e79-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963791    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.963910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963868    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.964117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.963917    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.964534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.964448    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.964719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.964656    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.964817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.964752    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.965473 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.965443    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91d447b-7696-4a75-9cdd-56dd77134e79-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.965572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.965525    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.966498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.966477    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:05.966612 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:05.966593    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:06.016401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:06.016374    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:06.016401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:06.016387    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8hs\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-kube-api-access-vp8hs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:06.309778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:06.308436    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:06.790121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:06.790097    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"]
Nov 03 12:17:06.792320 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:17:06.792287    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91d447b_7696_4a75_9cdd_56dd77134e79.slice/crio-bae71a0ca31436115191344a4469410dd950b36ac9ab5511adbd41922e72cf36 WatchSource:0}: Error finding container bae71a0ca31436115191344a4469410dd950b36ac9ab5511adbd41922e72cf36: Status 404 returned error can't find the container with id bae71a0ca31436115191344a4469410dd950b36ac9ab5511adbd41922e72cf36
Nov 03 12:17:06.794796 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:06.794762    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:06.794883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:06.794828    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:06.794883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:06.794859    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:07.209397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:07.209359    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" event={"ID":"d91d447b-7696-4a75-9cdd-56dd77134e79","Type":"ContainerStarted","Data":"c38b049e54cbd6cec887242123d90abaf4a03866bf37e5f55c9aff3383df2a82"}
Nov 03 12:17:07.209588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:07.209416    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" event={"ID":"d91d447b-7696-4a75-9cdd-56dd77134e79","Type":"ContainerStarted","Data":"bae71a0ca31436115191344a4469410dd950b36ac9ab5511adbd41922e72cf36"}
Nov 03 12:17:07.320724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:07.318809    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:07.320919 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:07.320860    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 12:17:07.321045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:07.320924    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 12:17:07.370817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:07.370754    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podStartSLOduration=2.370736231 podStartE2EDuration="2.370736231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:07.369286028 +0000 UTC m=+1576.381752693" watchObservedRunningTime="2026-11-03 12:17:07.370736231 +0000 UTC m=+1576.383202896"
Nov 03 12:17:08.309162 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:08.309126    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 12:17:08.309661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:08.309186    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 12:17:09.309067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:09.309035    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 12:17:09.309208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:09.309085    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 12:17:09.592372 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:17:09.592289    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:17:10.310040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:10.310007    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 12:17:10.310445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:10.310057    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 12:17:11.274817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.274729    4909 generic.go:356] "Generic (PLEG): container finished" podID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerID="04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b" exitCode=0
Nov 03 12:17:11.274817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.274770    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" event={"ID":"995db966-94b9-4691-a0ec-c5e88b54d7bb","Type":"ContainerDied","Data":"04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b"}
Nov 03 12:17:11.275327 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.275304    4909 scope.go:122] "RemoveContainer" containerID="04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b"
Nov 03 12:17:11.276933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.276908    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:11.277013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.276974    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:11.277013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.277005    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:11.309395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.309368    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 12:17:11.309494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:11.309417    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 12:17:12.197659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:12.197622    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:17:12.303673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:12.298127    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" event={"ID":"995db966-94b9-4691-a0ec-c5e88b54d7bb","Type":"ContainerStarted","Data":"0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808"}
Nov 03 12:17:12.309083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:12.309047    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 12:17:12.309218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:12.309103    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 12:17:12.511378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:12.511287    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:17:12.512408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:12.512382    4909 patch_prober.go:29] interesting pod/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 12:17:12.512544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:12.512423    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 12:17:13.309417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:13.309385    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 12:17:13.309820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:13.309436    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 12:17:13.512079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:13.512036    4909 patch_prober.go:29] interesting pod/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 12:17:13.512252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:13.512107    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 12:17:14.136724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:14.135787    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"]
Nov 03 12:17:14.511551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:14.511518    4909 patch_prober.go:29] interesting pod/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 12:17:14.511961 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:14.511568    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 12:17:14.757762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:14.756897    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:14.758483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:14.758458    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:14.792089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:14.792004    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:17:15.518721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:15.518069    4909 patch_prober.go:29] interesting pod/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 12:17:15.518721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:15.518134    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 12:17:16.511597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:16.511564    4909 patch_prober.go:29] interesting pod/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 12:17:16.511786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:16.511625    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 12:17:17.511291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:17.511248    4909 patch_prober.go:29] interesting pod/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 12:17:17.511789 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:17.511329    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 12:17:18.515593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:18.515563    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:17:18.516047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:18.515836    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:17:18.516610 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:18.516590    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:17:19.137046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.137003    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5" podUID="999ebb5e-bb1c-4096-8be7-7308676cd30c" containerName="payload-processing" containerID="cri-o://b4c1954230b0f9504945c13efc3f4102b3c6e6a880fe6fd5000dd5bff6732cc8" gracePeriod=25
Nov 03 12:17:19.350811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.350782    4909 generic.go:356] "Generic (PLEG): container finished" podID="999ebb5e-bb1c-4096-8be7-7308676cd30c" containerID="b4c1954230b0f9504945c13efc3f4102b3c6e6a880fe6fd5000dd5bff6732cc8" exitCode=0
Nov 03 12:17:19.350972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.350861    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5" event={"ID":"999ebb5e-bb1c-4096-8be7-7308676cd30c","Type":"ContainerDied","Data":"b4c1954230b0f9504945c13efc3f4102b3c6e6a880fe6fd5000dd5bff6732cc8"}
Nov 03 12:17:19.411192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.411169    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:17:19.504782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.504736    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999ebb5e-bb1c-4096-8be7-7308676cd30c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999ebb5e-bb1c-4096-8be7-7308676cd30c-plugins-config-volume\") pod \"999ebb5e-bb1c-4096-8be7-7308676cd30c\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") "
Nov 03 12:17:19.504956 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.504800    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999ebb5e-bb1c-4096-8be7-7308676cd30c-kube-api-access-bnvth\" (UniqueName: \"kubernetes.io/projected/999ebb5e-bb1c-4096-8be7-7308676cd30c-kube-api-access-bnvth\") pod \"999ebb5e-bb1c-4096-8be7-7308676cd30c\" (UID: \"999ebb5e-bb1c-4096-8be7-7308676cd30c\") "
Nov 03 12:17:19.505090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.505068    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999ebb5e-bb1c-4096-8be7-7308676cd30c-plugins-config-volume" pod "999ebb5e-bb1c-4096-8be7-7308676cd30c" (UID: "999ebb5e-bb1c-4096-8be7-7308676cd30c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:19.505636 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.505617    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999ebb5e-bb1c-4096-8be7-7308676cd30c-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:19.506952 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.506930    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999ebb5e-bb1c-4096-8be7-7308676cd30c-kube-api-access-bnvth" pod "999ebb5e-bb1c-4096-8be7-7308676cd30c" (UID: "999ebb5e-bb1c-4096-8be7-7308676cd30c"). InnerVolumeSpecName "kube-api-access-bnvth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:19.607106 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:19.607075    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvth\" (UniqueName: \"kubernetes.io/projected/999ebb5e-bb1c-4096-8be7-7308676cd30c-kube-api-access-bnvth\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:20.362519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:20.362474    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5" event={"ID":"999ebb5e-bb1c-4096-8be7-7308676cd30c","Type":"ContainerDied","Data":"cf5d701363d06288a78c090ff5858d27eebe39f3bcab799ba38d1c4edc81beba"}
Nov 03 12:17:20.362519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:20.362521    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"
Nov 03 12:17:20.362846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:20.362562    4909 scope.go:122] "RemoveContainer" containerID="b4c1954230b0f9504945c13efc3f4102b3c6e6a880fe6fd5000dd5bff6732cc8"
Nov 03 12:17:20.519004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:20.518963    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"]
Nov 03 12:17:20.563605 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:20.563563    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86d7ad-8fb688f5f-ljrx5"]
Nov 03 12:17:21.179564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:21.179528    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999ebb5e-bb1c-4096-8be7-7308676cd30c" path="/var/lib/kubelet/pods/999ebb5e-bb1c-4096-8be7-7308676cd30c/volumes"
Nov 03 12:17:31.516799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.516768    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"]
Nov 03 12:17:31.517513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.517493    4909 cpu_manager.go:414] "containerMap: removing container" podUID="999ebb5e-bb1c-4096-8be7-7308676cd30c" containerName="payload-processing"
Nov 03 12:17:31.517588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.517515    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999ebb5e-bb1c-4096-8be7-7308676cd30c" containerName="payload-processing"
Nov 03 12:17:31.517755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.517738    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="999ebb5e-bb1c-4096-8be7-7308676cd30c" containerName="payload-processing"
Nov 03 12:17:31.522719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.522682    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.616609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.616568    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"]
Nov 03 12:17:31.620419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.620395    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-89a88d-openshift-default-dockercfg-96hps\""
Nov 03 12:17:31.641725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.641670    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-certs\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.641862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.641722    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4c5d\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-kube-api-access-s4c5d\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.641862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.641814    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-credential-socket\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.641956 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.641869    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3340a2-3347-4399-9535-4443015fc780-istiod-ca-cert\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.642013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.641986    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-istio-token\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.642074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.642056    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-data\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.642242 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.642219    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-socket\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.642454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.642436    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3340a2-3347-4399-9535-4443015fc780-istio-podinfo\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.642538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.642518    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-envoy\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.774773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.745205    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-socket\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.775288 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.775247    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3340a2-3347-4399-9535-4443015fc780-istio-podinfo\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.775515 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.775495    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-envoy\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.775654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.775637    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-certs\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.775799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.775783    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c5d\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-kube-api-access-s4c5d\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.775911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.775896    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-credential-socket\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.776074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.776059    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3340a2-3347-4399-9535-4443015fc780-istiod-ca-cert\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.776212 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.776194    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-istio-token\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.776338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.776323    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-data\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.777071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.777045    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-data\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.785805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.782069    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-certs\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.785999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.770428    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-socket\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.786104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.783463    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3340a2-3347-4399-9535-4443015fc780-istiod-ca-cert\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.786197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.783728    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-credential-socket\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.814924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.814878    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3340a2-3347-4399-9535-4443015fc780-istio-podinfo\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.845752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.828405    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-envoy\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.872681 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.866116    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4c5d\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-kube-api-access-s4c5d\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:31.872918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:31.870160    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-istio-token\") pod \"e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") " pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:32.136291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:32.136203    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:32.555869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:32.555840    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"]
Nov 03 12:17:32.556335 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:17:32.556266    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3340a2_3347_4399_9535_4443015fc780.slice/crio-dde007e0a1e5ce576a487b964bcbfdc425aa9d406001cfd9cac481f19159a28f WatchSource:0}: Error finding container dde007e0a1e5ce576a487b964bcbfdc425aa9d406001cfd9cac481f19159a28f: Status 404 returned error can't find the container with id dde007e0a1e5ce576a487b964bcbfdc425aa9d406001cfd9cac481f19159a28f
Nov 03 12:17:32.559208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:32.559177    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:32.559344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:32.559241    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:32.559344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:32.559275    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:17:33.482528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:33.482485    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" event={"ID":"7b3340a2-3347-4399-9535-4443015fc780","Type":"ContainerStarted","Data":"aa3f73bb30b8570f8dc9be50d5e58c11771291a6a8bdcdf114fe03af9d92707a"}
Nov 03 12:17:33.482672 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:33.482609    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" event={"ID":"7b3340a2-3347-4399-9535-4443015fc780","Type":"ContainerStarted","Data":"dde007e0a1e5ce576a487b964bcbfdc425aa9d406001cfd9cac481f19159a28f"}
Nov 03 12:17:33.639506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:33.639457    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" podStartSLOduration=2.639443157 podStartE2EDuration="2.639443157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:33.591197255 +0000 UTC m=+1602.603663917" watchObservedRunningTime="2026-11-03 12:17:33.639443157 +0000 UTC m=+1602.651909879"
Nov 03 12:17:33.759776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:33.756077    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"]
Nov 03 12:17:33.759776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:33.756313    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9" podUID="c141b778-28d5-44b8-bc3c-5b762195ffb6" containerName="maas-api" containerID="cri-o://a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313" gracePeriod=30
Nov 03 12:17:34.136877 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.136785    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:34.144347 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.144296    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:34.341807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.338975    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:17:34.395458 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.395376    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-metrics-certs\") pod \"c141b778-28d5-44b8-bc3c-5b762195ffb6\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") "
Nov 03 12:17:34.395458 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.395441    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c141b778-28d5-44b8-bc3c-5b762195ffb6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c141b778-28d5-44b8-bc3c-5b762195ffb6-openshift-service-ca\") pod \"c141b778-28d5-44b8-bc3c-5b762195ffb6\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") "
Nov 03 12:17:34.395458 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.395464    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c141b778-28d5-44b8-bc3c-5b762195ffb6-kube-api-access-g7ntx\" (UniqueName: \"kubernetes.io/projected/c141b778-28d5-44b8-bc3c-5b762195ffb6-kube-api-access-g7ntx\") pod \"c141b778-28d5-44b8-bc3c-5b762195ffb6\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") "
Nov 03 12:17:34.395872 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.395652    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-maas-api-tls\") pod \"c141b778-28d5-44b8-bc3c-5b762195ffb6\" (UID: \"c141b778-28d5-44b8-bc3c-5b762195ffb6\") "
Nov 03 12:17:34.401461 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.401151    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c141b778-28d5-44b8-bc3c-5b762195ffb6-openshift-service-ca" pod "c141b778-28d5-44b8-bc3c-5b762195ffb6" (UID: "c141b778-28d5-44b8-bc3c-5b762195ffb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:34.408743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.407516    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-maas-api-tls" pod "c141b778-28d5-44b8-bc3c-5b762195ffb6" (UID: "c141b778-28d5-44b8-bc3c-5b762195ffb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:34.408743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.407857    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-metrics-certs" pod "c141b778-28d5-44b8-bc3c-5b762195ffb6" (UID: "c141b778-28d5-44b8-bc3c-5b762195ffb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:34.408743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.408101    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c141b778-28d5-44b8-bc3c-5b762195ffb6-kube-api-access-g7ntx" pod "c141b778-28d5-44b8-bc3c-5b762195ffb6" (UID: "c141b778-28d5-44b8-bc3c-5b762195ffb6"). InnerVolumeSpecName "kube-api-access-g7ntx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:34.491444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.491407    4909 generic.go:356] "Generic (PLEG): container finished" podID="c141b778-28d5-44b8-bc3c-5b762195ffb6" containerID="a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313" exitCode=0
Nov 03 12:17:34.491674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.491456    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"
Nov 03 12:17:34.491674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.491491    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9" event={"ID":"c141b778-28d5-44b8-bc3c-5b762195ffb6","Type":"ContainerDied","Data":"a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313"}
Nov 03 12:17:34.491674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.491553    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9" event={"ID":"c141b778-28d5-44b8-bc3c-5b762195ffb6","Type":"ContainerDied","Data":"ad9f3b61b95aadaf76e14cb46ec4d60acb1a948e3a34f7d1094611fb0e683a69"}
Nov 03 12:17:34.491674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.491584    4909 scope.go:122] "RemoveContainer" containerID="a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313"
Nov 03 12:17:34.492576 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.492511    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:34.493428 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.493411    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:17:34.497861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.497840    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:34.498001 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.497984    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c141b778-28d5-44b8-bc3c-5b762195ffb6-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:34.498102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.498089    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ntx\" (UniqueName: \"kubernetes.io/projected/c141b778-28d5-44b8-bc3c-5b762195ffb6-kube-api-access-g7ntx\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:34.498197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.498185    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c141b778-28d5-44b8-bc3c-5b762195ffb6-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:34.507954 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.507936    4909 scope.go:122] "RemoveContainer" containerID="a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313"
Nov 03 12:17:34.508234 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:17:34.508207    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313\": container with ID starting with a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313 not found: ID does not exist" containerID="a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313"
Nov 03 12:17:34.508326 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.508244    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313"} err="failed to get container status \"a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313\": rpc error: code = NotFound desc = could not find container \"a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313\": container with ID starting with a246753361f9eb45b3693328832d48844ca2fa3d72bc185693ff3ae39c273313 not found: ID does not exist"
Nov 03 12:17:34.649485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.649406    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"]
Nov 03 12:17:34.653547 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.653519    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac6071-7b757cbd9c-z9nj9"]
Nov 03 12:17:34.915324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.915242    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"]
Nov 03 12:17:34.915909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.915893    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c141b778-28d5-44b8-bc3c-5b762195ffb6" containerName="maas-api"
Nov 03 12:17:34.915909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.915910    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c141b778-28d5-44b8-bc3c-5b762195ffb6" containerName="maas-api"
Nov 03 12:17:34.916050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.916008    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c141b778-28d5-44b8-bc3c-5b762195ffb6" containerName="maas-api"
Nov 03 12:17:34.920579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.920564    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:34.990732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.989850    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8e8d12\""
Nov 03 12:17:34.997722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:34.991083    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8e8d12-dockercfg-v7gkm\""
Nov 03 12:17:35.018722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.012582    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95f8\" (UniqueName: \"kubernetes.io/projected/0ac230b1-3a43-428f-bad7-10795aa2bd7c-kube-api-access-l95f8\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:35.018722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.012655    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac230b1-3a43-428f-bad7-10795aa2bd7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:35.038135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.038101    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"]
Nov 03 12:17:35.038382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.038353    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7" podUID="f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" containerName="maas-api" containerID="cri-o://ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d" gracePeriod=30
Nov 03 12:17:35.119087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.118805    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac230b1-3a43-428f-bad7-10795aa2bd7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:35.119290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.119242    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95f8\" (UniqueName: \"kubernetes.io/projected/0ac230b1-3a43-428f-bad7-10795aa2bd7c-kube-api-access-l95f8\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:35.159749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.120629    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac230b1-3a43-428f-bad7-10795aa2bd7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:35.251723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.234999    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95f8\" (UniqueName: \"kubernetes.io/projected/0ac230b1-3a43-428f-bad7-10795aa2bd7c-kube-api-access-l95f8\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:35.251723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.235238    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:35.412724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.395585    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c141b778-28d5-44b8-bc3c-5b762195ffb6" path="/var/lib/kubelet/pods/c141b778-28d5-44b8-bc3c-5b762195ffb6/volumes"
Nov 03 12:17:35.412724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.396053    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"]
Nov 03 12:17:35.864872 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.864849    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:17:36.007737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:35.990550    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"]
Nov 03 12:17:36.025727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.013307    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-kube-api-access-lzrqt\" (UniqueName: \"kubernetes.io/projected/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-kube-api-access-lzrqt\") pod \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") "
Nov 03 12:17:36.025727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.013415    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-maas-api-tls\") pod \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") "
Nov 03 12:17:36.025727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.013478    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-openshift-service-ca\") pod \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") "
Nov 03 12:17:36.025727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.013617    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-metrics-certs\") pod \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\" (UID: \"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2\") "
Nov 03 12:17:36.027420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.027385    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-openshift-service-ca" pod "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" (UID: "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:36.055923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.055882    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-maas-api-tls" pod "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" (UID: "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:36.079089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.079044    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-metrics-certs" pod "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" (UID: "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:36.129057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.128978    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:36.129057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.129017    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:36.129057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.129032    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:36.144099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.144072    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-kube-api-access-lzrqt" pod "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" (UID: "f1b7517f-8ab0-45f7-990b-f2636ddb5fe2"). InnerVolumeSpecName "kube-api-access-lzrqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:36.230430 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.230406    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzrqt\" (UniqueName: \"kubernetes.io/projected/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2-kube-api-access-lzrqt\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:36.543654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.543619    4909 generic.go:356] "Generic (PLEG): container finished" podID="f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" containerID="ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d" exitCode=0
Nov 03 12:17:36.543889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.543670    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"
Nov 03 12:17:36.543889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.543731    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7" event={"ID":"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2","Type":"ContainerDied","Data":"ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d"}
Nov 03 12:17:36.543889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.543794    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7" event={"ID":"f1b7517f-8ab0-45f7-990b-f2636ddb5fe2","Type":"ContainerDied","Data":"28e192511173e0d729330eb2a4502dc250224090a563b31bf09614c4a7afa1ef"}
Nov 03 12:17:36.543889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.543825    4909 scope.go:122] "RemoveContainer" containerID="ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d"
Nov 03 12:17:36.545511 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.545477    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c" event={"ID":"0ac230b1-3a43-428f-bad7-10795aa2bd7c","Type":"ContainerStarted","Data":"fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e"}
Nov 03 12:17:36.545607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.545539    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c" event={"ID":"0ac230b1-3a43-428f-bad7-10795aa2bd7c","Type":"ContainerStarted","Data":"1f596ae0af95816e8c14fe7e59715376f5dfda479d19e51e0d10c35d96398f73"}
Nov 03 12:17:36.545746 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.545731    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:36.553879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.553863    4909 scope.go:122] "RemoveContainer" containerID="ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d"
Nov 03 12:17:36.554131 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:17:36.554115    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d\": container with ID starting with ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d not found: ID does not exist" containerID="ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d"
Nov 03 12:17:36.554172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.554139    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d"} err="failed to get container status \"ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d\": rpc error: code = NotFound desc = could not find container \"ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d\": container with ID starting with ebfed2a8eee7da2e23cb09165f717bb04f4dc773ec2a9e9e409f88b3da1e595d not found: ID does not exist"
Nov 03 12:17:36.738721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.725330    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c" podStartSLOduration=2.725309641 podStartE2EDuration="2.725309641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:36.724079969 +0000 UTC m=+1605.736546633" watchObservedRunningTime="2026-11-03 12:17:36.725309641 +0000 UTC m=+1605.737776305"
Nov 03 12:17:36.846943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.846858    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"]
Nov 03 12:17:36.864721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:36.862814    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86d7ad-7c4bf8975c-8kpq7"]
Nov 03 12:17:37.124482 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:37.124412    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" path="/var/lib/kubelet/pods/f1b7517f-8ab0-45f7-990b-f2636ddb5fe2/volumes"
Nov 03 12:17:39.751498 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:17:39.747954    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:17:39.878720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.878599    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"]
Nov 03 12:17:39.885725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.885666    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" containerName="maas-api"
Nov 03 12:17:39.885941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.885921    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" containerName="maas-api"
Nov 03 12:17:39.886559 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.886539    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b7517f-8ab0-45f7-990b-f2636ddb5fe2" containerName="maas-api"
Nov 03 12:17:39.895038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.895005    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:39.960298 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.959866    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"]
Nov 03 12:17:39.967399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.967372    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-89a88d\""
Nov 03 12:17:39.967576 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:39.967449    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-89a88d-dockercfg-th492\""
Nov 03 12:17:40.055786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.054570    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"]
Nov 03 12:17:40.222031 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.161995    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60313337-ed50-4185-9a57-3e162f277efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:40.260510 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.259151    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94j8k\" (UniqueName: \"kubernetes.io/projected/60313337-ed50-4185-9a57-3e162f277efb-kube-api-access-94j8k\") pod \"payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:40.317982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.317906    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"]
Nov 03 12:17:40.318194 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.318074    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.375186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.375158    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94j8k\" (UniqueName: \"kubernetes.io/projected/60313337-ed50-4185-9a57-3e162f277efb-kube-api-access-94j8k\") pod \"payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:40.375374 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.375352    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdvw\" (UniqueName: \"kubernetes.io/projected/73188b29-c22c-413f-8569-1cc7e0c738e6-kube-api-access-bcdvw\") pod \"payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") " pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.375445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.375407    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73188b29-c22c-413f-8569-1cc7e0c738e6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") " pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.375681 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.375654    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60313337-ed50-4185-9a57-3e162f277efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:40.376240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.376222    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60313337-ed50-4185-9a57-3e162f277efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:40.427579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.427544    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94j8k\" (UniqueName: \"kubernetes.io/projected/60313337-ed50-4185-9a57-3e162f277efb-kube-api-access-94j8k\") pod \"payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:40.476924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.476896    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdvw\" (UniqueName: \"kubernetes.io/projected/73188b29-c22c-413f-8569-1cc7e0c738e6-kube-api-access-bcdvw\") pod \"payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") " pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.477052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.476930    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73188b29-c22c-413f-8569-1cc7e0c738e6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") " pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.477661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.477642    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73188b29-c22c-413f-8569-1cc7e0c738e6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") " pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.529008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.528986    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdvw\" (UniqueName: \"kubernetes.io/projected/73188b29-c22c-413f-8569-1cc7e0c738e6-kube-api-access-bcdvw\") pod \"payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") " pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.569217 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.569151    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:40.639412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.639371    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:40.935360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:40.935333    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"]
Nov 03 12:17:40.935749 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:17:40.935640    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60313337_ed50_4185_9a57_3e162f277efb.slice/crio-c7051baf7a875f86962de89d9fd9d641d77dcd3e67d10812740bd9da85b3937a WatchSource:0}: Error finding container c7051baf7a875f86962de89d9fd9d641d77dcd3e67d10812740bd9da85b3937a: Status 404 returned error can't find the container with id c7051baf7a875f86962de89d9fd9d641d77dcd3e67d10812740bd9da85b3937a
Nov 03 12:17:41.027680 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.027657    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"]
Nov 03 12:17:41.040358 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:17:41.040331    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73188b29_c22c_413f_8569_1cc7e0c738e6.slice/crio-334056a51056b43f9a510408d8969d2969edf370df360a03b0c5dabf1a096207 WatchSource:0}: Error finding container 334056a51056b43f9a510408d8969d2969edf370df360a03b0c5dabf1a096207: Status 404 returned error can't find the container with id 334056a51056b43f9a510408d8969d2969edf370df360a03b0c5dabf1a096207
Nov 03 12:17:41.729457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.722814    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb" event={"ID":"73188b29-c22c-413f-8569-1cc7e0c738e6","Type":"ContainerStarted","Data":"45b9426da44fca678967419616cb6482bf257e9055bccd4446ae9128ed212992"}
Nov 03 12:17:41.729457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.722884    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb" event={"ID":"73188b29-c22c-413f-8569-1cc7e0c738e6","Type":"ContainerStarted","Data":"334056a51056b43f9a510408d8969d2969edf370df360a03b0c5dabf1a096207"}
Nov 03 12:17:41.729457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.724031    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:41.729457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.726265    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4" event={"ID":"60313337-ed50-4185-9a57-3e162f277efb","Type":"ContainerStarted","Data":"2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905"}
Nov 03 12:17:41.729457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.726311    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4" event={"ID":"60313337-ed50-4185-9a57-3e162f277efb","Type":"ContainerStarted","Data":"c7051baf7a875f86962de89d9fd9d641d77dcd3e67d10812740bd9da85b3937a"}
Nov 03 12:17:41.729457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.727047    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:41.827932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.827879    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb" podStartSLOduration=2.827862388 podStartE2EDuration="2.827862388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:41.827817464 +0000 UTC m=+1610.840284128" watchObservedRunningTime="2026-11-03 12:17:41.827862388 +0000 UTC m=+1610.840329051"
Nov 03 12:17:41.935301 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:41.935118    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4" podStartSLOduration=2.935102126 podStartE2EDuration="2.935102126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:41.931857757 +0000 UTC m=+1610.944324420" watchObservedRunningTime="2026-11-03 12:17:41.935102126 +0000 UTC m=+1610.947568790"
Nov 03 12:17:42.213853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:42.213827    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:17:44.585206 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:44.585166    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"]
Nov 03 12:17:44.585781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:44.585426    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy" containerID="cri-o://db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05" gracePeriod=30
Nov 03 12:17:47.556925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:47.556896    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:17:49.861635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.861577    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:17:49.972738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.972710    4909 generic.go:356] "Generic (PLEG): container finished" podID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerID="db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05" exitCode=0
Nov 03 12:17:49.972874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.972758    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"
Nov 03 12:17:49.972874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.972792    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" event={"ID":"d8cb4163-c911-4f2b-97f8-4643635d95c7","Type":"ContainerDied","Data":"db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05"}
Nov 03 12:17:49.972874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.972855    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p" event={"ID":"d8cb4163-c911-4f2b-97f8-4643635d95c7","Type":"ContainerDied","Data":"3cf2237fe2cf4f10d2c2cfcde2f4bdd778a71c8ff2ca5b3241e2ad4127254628"}
Nov 03 12:17:49.973000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.972881    4909 scope.go:122] "RemoveContainer" containerID="db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05"
Nov 03 12:17:49.979267 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979223    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-podinfo\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.979377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979316    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-socket\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.979421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979385    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-envoy\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.979785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979540    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-credential-socket\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.979785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979619    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-data\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.979785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979655    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-token\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-token\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.979785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979724    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-certs\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.980051 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979804    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8cb4163-c911-4f2b-97f8-4643635d95c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cb4163-c911-4f2b-97f8-4643635d95c7-istiod-ca-cert\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.980051 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.979831    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-kube-api-access-dmrq5\") pod \"d8cb4163-c911-4f2b-97f8-4643635d95c7\" (UID: \"d8cb4163-c911-4f2b-97f8-4643635d95c7\") "
Nov 03 12:17:49.980461 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.980437    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-data" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:49.981484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.981377    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-socket" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:49.982196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.982170    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-certs" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:49.982306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.982186    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-credential-socket" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:49.982373 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.982315    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8cb4163-c911-4f2b-97f8-4643635d95c7-istiod-ca-cert" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:49.982788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.982766    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-kube-api-access-dmrq5" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "kube-api-access-dmrq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:49.983572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.983547    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-token" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:49.983824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.983801    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-podinfo" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:17:49.983824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.983817    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-envoy" pod "d8cb4163-c911-4f2b-97f8-4643635d95c7" (UID: "d8cb4163-c911-4f2b-97f8-4643635d95c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:49.984931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:49.984909    4909 scope.go:122] "RemoveContainer" containerID="d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a"
Nov 03 12:17:50.010532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.010511    4909 scope.go:122] "RemoveContainer" containerID="db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05"
Nov 03 12:17:50.010830 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:17:50.010810    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05\": container with ID starting with db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05 not found: ID does not exist" containerID="db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05"
Nov 03 12:17:50.010887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.010841    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05"} err="failed to get container status \"db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05\": rpc error: code = NotFound desc = could not find container \"db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05\": container with ID starting with db7022aa734c061abac9f23e363835983c109cb2eeb443efa4ab19cd19cf1f05 not found: ID does not exist"
Nov 03 12:17:50.010887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.010860    4909 scope.go:122] "RemoveContainer" containerID="d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a"
Nov 03 12:17:50.011133 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:17:50.011109    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a\": container with ID starting with d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a not found: ID does not exist" containerID="d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a"
Nov 03 12:17:50.011198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.011144    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a"} err="failed to get container status \"d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a\": rpc error: code = NotFound desc = could not find container \"d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a\": container with ID starting with d03c8ce365d53405121d31720a14a62736906faa1e8c1cba06b683c208af9c0a not found: ID does not exist"
Nov 03 12:17:50.082090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082053    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082084    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082095    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082104    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082113    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cb4163-c911-4f2b-97f8-4643635d95c7-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082122    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/d8cb4163-c911-4f2b-97f8-4643635d95c7-kube-api-access-dmrq5\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082130    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082139    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.082371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.082147    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cb4163-c911-4f2b-97f8-4643635d95c7-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:50.387072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.387037    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"]
Nov 03 12:17:50.432846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:50.432817    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ac6071-openshift-default-7694479bdf-drf4p"]
Nov 03 12:17:51.123779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:51.123750    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" path="/var/lib/kubelet/pods/d8cb4163-c911-4f2b-97f8-4643635d95c7/volumes"
Nov 03 12:17:53.937187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:53.937158    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:17:53.937587 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:53.937309    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:17:54.199519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:54.199491    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"]
Nov 03 12:17:54.199781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:54.199751    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy" containerID="cri-o://090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0" gracePeriod=30
Nov 03 12:17:59.493421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.493391    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:17:59.660965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.660932    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-credential-socket\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661009    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-certs\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661056    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istiod-ca-cert\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661130    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-envoy\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661184    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-kube-api-access-dpskc\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-kube-api-access-dpskc\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661237    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-credential-socket" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:59.661308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661272    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-socket\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661308    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-podinfo\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661367    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-data\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.661574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.661517    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-token\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-token\") pod \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\" (UID: \"090e61b1-c21b-4fe9-9ca0-7e536abdd75b\") "
Nov 03 12:17:59.662554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.662522    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-socket" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:59.663002 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.662974    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istiod-ca-cert" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:59.663447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.663383    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.663447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.663412    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.663447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.663426    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.663820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.663786    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-data" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:59.664083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.664050    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-kube-api-access-dpskc" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "kube-api-access-dpskc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:59.664176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.664116    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-token" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:59.664247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.664169    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-certs" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:59.666234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.666207    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-envoy" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:59.666330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.666284    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-podinfo" pod "090e61b1-c21b-4fe9-9ca0-7e536abdd75b" (UID: "090e61b1-c21b-4fe9-9ca0-7e536abdd75b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:17:59.769853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.769665    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.769853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.769747    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.769853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.769768    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.769853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.769791    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.769853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.769809    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.769853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:17:59.769829    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpskc\" (UniqueName: \"kubernetes.io/projected/090e61b1-c21b-4fe9-9ca0-7e536abdd75b-kube-api-access-dpskc\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:00.075006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.074927    4909 generic.go:356] "Generic (PLEG): container finished" podID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerID="090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0" exitCode=0
Nov 03 12:18:00.075006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.074975    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"
Nov 03 12:18:00.075200 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.075000    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" event={"ID":"090e61b1-c21b-4fe9-9ca0-7e536abdd75b","Type":"ContainerDied","Data":"090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0"}
Nov 03 12:18:00.075200 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.075059    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx" event={"ID":"090e61b1-c21b-4fe9-9ca0-7e536abdd75b","Type":"ContainerDied","Data":"435cc39f6b32f05809cd857fa4909f296026fc9f390fdde3076a69d348db3451"}
Nov 03 12:18:00.075200 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.075082    4909 scope.go:122] "RemoveContainer" containerID="090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0"
Nov 03 12:18:00.086843 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.086820    4909 scope.go:122] "RemoveContainer" containerID="090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0"
Nov 03 12:18:00.087130 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:00.087108    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0\": container with ID starting with 090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0 not found: ID does not exist" containerID="090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0"
Nov 03 12:18:00.087212 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.087138    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0"} err="failed to get container status \"090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0\": rpc error: code = NotFound desc = could not find container \"090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0\": container with ID starting with 090fa204a280ad8cde86bcea42816f3251b0dd2837acb8e6db3143d5958612a0 not found: ID does not exist"
Nov 03 12:18:00.229591 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.229541    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"]
Nov 03 12:18:00.277056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:00.277028    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-bcdtx"]
Nov 03 12:18:01.124871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:01.124839    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" path="/var/lib/kubelet/pods/090e61b1-c21b-4fe9-9ca0-7e536abdd75b/volumes"
Nov 03 12:18:03.001771 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.001738    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"]
Nov 03 12:18:03.002529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.002508    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy"
Nov 03 12:18:03.002529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.002529    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy"
Nov 03 12:18:03.002726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.002583    4909 cpu_manager.go:414] "containerMap: removing container" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy"
Nov 03 12:18:03.002726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.002591    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy"
Nov 03 12:18:03.002840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.002776    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="090e61b1-c21b-4fe9-9ca0-7e536abdd75b" containerName="istio-proxy"
Nov 03 12:18:03.002840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.002797    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy"
Nov 03 12:18:03.002840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.002812    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy"
Nov 03 12:18:03.007895 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.007871    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.037638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.037585    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpgl\" (UniqueName: \"kubernetes.io/projected/dd899088-3994-4cdc-bca3-97bd4cda35e9-kube-api-access-hzpgl\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.038107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.038025    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd899088-3994-4cdc-bca3-97bd4cda35e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.077927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.077887    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"]
Nov 03 12:18:03.116797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.109106    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-69fd8043\""
Nov 03 12:18:03.116797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.109295    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-69fd8043-dockercfg-vlxp6\""
Nov 03 12:18:03.156725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.140410    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd899088-3994-4cdc-bca3-97bd4cda35e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.160852 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.160821    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpgl\" (UniqueName: \"kubernetes.io/projected/dd899088-3994-4cdc-bca3-97bd4cda35e9-kube-api-access-hzpgl\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.168340 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.155443    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd899088-3994-4cdc-bca3-97bd4cda35e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.249518 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.249480    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpgl\" (UniqueName: \"kubernetes.io/projected/dd899088-3994-4cdc-bca3-97bd4cda35e9-kube-api-access-hzpgl\") pod \"payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") " pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.322485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.322400    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:03.691161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:03.691128    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"]
Nov 03 12:18:03.692250 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:18:03.692224    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd899088_3994_4cdc_bca3_97bd4cda35e9.slice/crio-fbda50ca79d165b643ba2ff9aad79cc3f9d3c27e9da2190456eeacc2ba05b01e WatchSource:0}: Error finding container fbda50ca79d165b643ba2ff9aad79cc3f9d3c27e9da2190456eeacc2ba05b01e: Status 404 returned error can't find the container with id fbda50ca79d165b643ba2ff9aad79cc3f9d3c27e9da2190456eeacc2ba05b01e
Nov 03 12:18:04.197025 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:04.196995    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb" event={"ID":"dd899088-3994-4cdc-bca3-97bd4cda35e9","Type":"ContainerStarted","Data":"af0bb0f669c7a12bb44b48705c8e2f38173632963811650ab71437c3f23c820e"}
Nov 03 12:18:04.197409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:04.197053    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:04.197409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:04.197072    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb" event={"ID":"dd899088-3994-4cdc-bca3-97bd4cda35e9","Type":"ContainerStarted","Data":"fbda50ca79d165b643ba2ff9aad79cc3f9d3c27e9da2190456eeacc2ba05b01e"}
Nov 03 12:18:04.302838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:04.302792    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb" podStartSLOduration=2.302779623 podStartE2EDuration="2.302779623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:04.301256472 +0000 UTC m=+1633.313723136" watchObservedRunningTime="2026-11-03 12:18:04.302779623 +0000 UTC m=+1633.315246285"
Nov 03 12:18:09.564683 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:09.564646    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:18:11.162775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.162738    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"]
Nov 03 12:18:11.163526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.163505    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy"
Nov 03 12:18:11.163526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.163526    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8cb4163-c911-4f2b-97f8-4643635d95c7" containerName="istio-proxy"
Nov 03 12:18:11.167145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.167125    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.217210 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.217188    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8e8d12-serving-cert\""
Nov 03 12:18:11.221541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.221517    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"]
Nov 03 12:18:11.282676 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.282646    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad58d7ea-9900-4ad0-a9b4-e303804b7031-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.282878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.282859    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.282981 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.282965    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8kh\" (UniqueName: \"kubernetes.io/projected/ad58d7ea-9900-4ad0-a9b4-e303804b7031-kube-api-access-gd8kh\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.283042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.282998    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.384311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.384274    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8kh\" (UniqueName: \"kubernetes.io/projected/ad58d7ea-9900-4ad0-a9b4-e303804b7031-kube-api-access-gd8kh\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.384492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.384323    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.384492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.384440    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad58d7ea-9900-4ad0-a9b4-e303804b7031-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.384610 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.384567    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.385135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.385109    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad58d7ea-9900-4ad0-a9b4-e303804b7031-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.386827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.386806    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.386914 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.386833    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.437097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.437038    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8kh\" (UniqueName: \"kubernetes.io/projected/ad58d7ea-9900-4ad0-a9b4-e303804b7031-kube-api-access-gd8kh\") pod \"maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.481071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.481047    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:11.887231 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:11.887202    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"]
Nov 03 12:18:11.889465 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:18:11.889439    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad58d7ea_9900_4ad0_a9b4_e303804b7031.slice/crio-3b2903351b8af10a24c8a53e3d479a1cca62cd5d51ae716f59e63bb1ee2ae6f7 WatchSource:0}: Error finding container 3b2903351b8af10a24c8a53e3d479a1cca62cd5d51ae716f59e63bb1ee2ae6f7: Status 404 returned error can't find the container with id 3b2903351b8af10a24c8a53e3d479a1cca62cd5d51ae716f59e63bb1ee2ae6f7
Nov 03 12:18:12.264884 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:12.264847    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv" event={"ID":"ad58d7ea-9900-4ad0-a9b4-e303804b7031","Type":"ContainerStarted","Data":"3b2903351b8af10a24c8a53e3d479a1cca62cd5d51ae716f59e63bb1ee2ae6f7"}
Nov 03 12:18:13.272285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:13.272246    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv" event={"ID":"ad58d7ea-9900-4ad0-a9b4-e303804b7031","Type":"ContainerStarted","Data":"6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea"}
Nov 03 12:18:13.272685 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:13.272320    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:13.380445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:13.380395    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv" podStartSLOduration=1.370345664 podStartE2EDuration="2.380381916s" totalImagesPullingTime="1.010036252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:13.378142885 +0000 UTC m=+1642.390609548" watchObservedRunningTime="2026-11-03 12:18:13.380381916 +0000 UTC m=+1642.392848624"
Nov 03 12:18:15.206436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:15.206405    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:18:19.284572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:19.284533    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:18:21.068812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:21.068777    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"]
Nov 03 12:18:21.069190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:21.068997    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy" containerID="cri-o://466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088" gracePeriod=30
Nov 03 12:18:21.115059 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:21.115030    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"]
Nov 03 12:18:21.115274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:21.115243    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" containerID="cri-o://542633bd06ff738e7bbe8ee7828430ba2cee52919a4eb9d9da216af239ac1f6c" gracePeriod=30
Nov 03 12:18:26.389094 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.389060    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:18:26.407574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.407549    4909 generic.go:356] "Generic (PLEG): container finished" podID="d20d810b-8252-4795-8142-9811c3381eb1" containerID="542633bd06ff738e7bbe8ee7828430ba2cee52919a4eb9d9da216af239ac1f6c" exitCode=0
Nov 03 12:18:26.407752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.407627    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" event={"ID":"d20d810b-8252-4795-8142-9811c3381eb1","Type":"ContainerDied","Data":"542633bd06ff738e7bbe8ee7828430ba2cee52919a4eb9d9da216af239ac1f6c"}
Nov 03 12:18:26.409030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.409005    4909 generic.go:356] "Generic (PLEG): container finished" podID="0025920d-a5a7-481f-97b3-84e86203ad72" containerID="466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088" exitCode=0
Nov 03 12:18:26.409164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.409032    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" event={"ID":"0025920d-a5a7-481f-97b3-84e86203ad72","Type":"ContainerDied","Data":"466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088"}
Nov 03 12:18:26.409164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.409070    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"
Nov 03 12:18:26.409164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.409145    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w" event={"ID":"0025920d-a5a7-481f-97b3-84e86203ad72","Type":"ContainerDied","Data":"3e40d8d7849f37df4951f2d354804e2ab92578e0ea7240f60aa495295b8d37a6"}
Nov 03 12:18:26.409302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.409164    4909 scope.go:122] "RemoveContainer" containerID="466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088"
Nov 03 12:18:26.432048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.432033    4909 scope.go:122] "RemoveContainer" containerID="466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088"
Nov 03 12:18:26.432288 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:26.432272    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088\": container with ID starting with 466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088 not found: ID does not exist" containerID="466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088"
Nov 03 12:18:26.432336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.432295    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088"} err="failed to get container status \"466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088\": rpc error: code = NotFound desc = could not find container \"466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088\": container with ID starting with 466bac5658d576aba24047efec3a9a673609de8c1a59e0a90ed56b85ca20c088 not found: ID does not exist"
Nov 03 12:18:26.435348 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.435331    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:18:26.461424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461398    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0025920d-a5a7-481f-97b3-84e86203ad72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0025920d-a5a7-481f-97b3-84e86203ad72-istio-podinfo\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461447    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0025920d-a5a7-481f-97b3-84e86203ad72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0025920d-a5a7-481f-97b3-84e86203ad72-istiod-ca-cert\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461531    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-kube-api-access-nf2rs\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-kube-api-access-nf2rs\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461563    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-envoy\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461682 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461609    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-certs\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461729    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-data\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461784    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-istio-token\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-istio-token\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461901 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461884    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-socket\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.461994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461973    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-data" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.461994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461985    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-credential-socket\") pod \"0025920d-a5a7-481f-97b3-84e86203ad72\" (UID: \"0025920d-a5a7-481f-97b3-84e86203ad72\") "
Nov 03 12:18:26.462303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.461908    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0025920d-a5a7-481f-97b3-84e86203ad72-istiod-ca-cert" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:26.462421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462188    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-socket" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.462421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462202    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-credential-socket" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.462421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462224    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-certs" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.462652 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462636    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0025920d-a5a7-481f-97b3-84e86203ad72-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.462689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462654    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.462689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462663    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.462689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462672    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.462689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.462680    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.464055 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.464035    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0025920d-a5a7-481f-97b3-84e86203ad72-istio-podinfo" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:18:26.464196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.464170    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-kube-api-access-nf2rs" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "kube-api-access-nf2rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:26.464273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.464202    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-envoy" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.464273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.464206    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-istio-token" pod "0025920d-a5a7-481f-97b3-84e86203ad72" (UID: "0025920d-a5a7-481f-97b3-84e86203ad72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:26.564033 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564009    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-istio-token\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-istio-token\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564049    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-data\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564082    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-socket\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564110    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d20d810b-8252-4795-8142-9811c3381eb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d810b-8252-4795-8142-9811c3381eb1-istio-podinfo\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564140    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-certs\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564167    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-credential-socket\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564232    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-envoy\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564263    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-kube-api-access-9h9jl\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-kube-api-access-9h9jl\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564319    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20d810b-8252-4795-8142-9811c3381eb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d810b-8252-4795-8142-9811c3381eb1-istiod-ca-cert\") pod \"d20d810b-8252-4795-8142-9811c3381eb1\" (UID: \"d20d810b-8252-4795-8142-9811c3381eb1\") "
Nov 03 12:18:26.564362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564333    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-data" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.564660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564446    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-socket" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.564660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564465    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-certs" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.564660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564478    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-credential-socket" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.564830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.564809    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20d810b-8252-4795-8142-9811c3381eb1-istiod-ca-cert" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:26.565104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565073    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0025920d-a5a7-481f-97b3-84e86203ad72-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565097    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565109    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565118    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf2rs\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-kube-api-access-nf2rs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565127    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0025920d-a5a7-481f-97b3-84e86203ad72-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565137    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565145    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565154    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d810b-8252-4795-8142-9811c3381eb1-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.565252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.565162    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0025920d-a5a7-481f-97b3-84e86203ad72-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.566411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.566388    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-kube-api-access-9h9jl" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "kube-api-access-9h9jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:26.566635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.566611    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-istio-token" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:26.566635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.566612    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d20d810b-8252-4795-8142-9811c3381eb1-istio-podinfo" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:18:26.566784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.566643    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-envoy" pod "d20d810b-8252-4795-8142-9811c3381eb1" (UID: "d20d810b-8252-4795-8142-9811c3381eb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:26.667265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.667035    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.667265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.667071    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d810b-8252-4795-8142-9811c3381eb1-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.667265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.667085    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d810b-8252-4795-8142-9811c3381eb1-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.667265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.667101    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h9jl\" (UniqueName: \"kubernetes.io/projected/d20d810b-8252-4795-8142-9811c3381eb1-kube-api-access-9h9jl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:26.826167 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.826132    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"]
Nov 03 12:18:26.829588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:26.829563    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-v8f7w"]
Nov 03 12:18:27.124176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.124148    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" path="/var/lib/kubelet/pods/0025920d-a5a7-481f-97b3-84e86203ad72/volumes"
Nov 03 12:18:27.376749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.376645    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:18:27.376749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.376732    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:18:27.416493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.416468    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"
Nov 03 12:18:27.416927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.416467    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2" event={"ID":"d20d810b-8252-4795-8142-9811c3381eb1","Type":"ContainerDied","Data":"b31a43be6cc53d952b0194b48f34c6b3b977215511eab099c70833728d6b0e22"}
Nov 03 12:18:27.416927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.416598    4909 scope.go:122] "RemoveContainer" containerID="542633bd06ff738e7bbe8ee7828430ba2cee52919a4eb9d9da216af239ac1f6c"
Nov 03 12:18:27.569761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.569730    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"]
Nov 03 12:18:27.615640 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:27.615607    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-65rb2"]
Nov 03 12:18:29.125400 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:29.125371    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20d810b-8252-4795-8142-9811c3381eb1" path="/var/lib/kubelet/pods/d20d810b-8252-4795-8142-9811c3381eb1/volumes"
Nov 03 12:18:34.079013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.078797    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:18:34.452734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.452690    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"]
Nov 03 12:18:34.453385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.453362    4909 cpu_manager.go:414] "containerMap: removing container" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy"
Nov 03 12:18:34.453385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.453382    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy"
Nov 03 12:18:34.453569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.453413    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy"
Nov 03 12:18:34.453569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.453418    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy"
Nov 03 12:18:34.453569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.453535    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="0025920d-a5a7-481f-97b3-84e86203ad72" containerName="istio-proxy"
Nov 03 12:18:34.453569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.453548    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20d810b-8252-4795-8142-9811c3381eb1" containerName="istio-proxy"
Nov 03 12:18:34.459049 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.459030    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.500052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.500025    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56sb\" (UniqueName: \"kubernetes.io/projected/b7db405b-a248-4f5d-8f00-15f9cd647c57-kube-api-access-d56sb\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.500184 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.500083    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.500280 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.500267    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-metrics-certs\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.500317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.500295    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db405b-a248-4f5d-8f00-15f9cd647c57-openshift-service-ca\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.512144 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.512117    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"]
Nov 03 12:18:34.552963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.552940    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-69fd8043-serving-cert\""
Nov 03 12:18:34.601450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.601420    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56sb\" (UniqueName: \"kubernetes.io/projected/b7db405b-a248-4f5d-8f00-15f9cd647c57-kube-api-access-d56sb\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.601614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.601506    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.601760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.601744    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-metrics-certs\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.601853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.601787    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db405b-a248-4f5d-8f00-15f9cd647c57-openshift-service-ca\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.601917 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:34.601886    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-serving-cert: secret "maas-api-e2e-mt-69fd8043-serving-cert" not found
Nov 03 12:18:34.601973 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:34.601949    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls podName:b7db405b-a248-4f5d-8f00-15f9cd647c57 nodeName:}" failed. No retries permitted until 2026-11-03 12:18:35.101928983 +0000 UTC m=+1664.114395626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls") pod "maas-api-e2e-mt-69fd8043-64675459d-54zkf" (UID: "b7db405b-a248-4f5d-8f00-15f9cd647c57") : secret "maas-api-e2e-mt-69fd8043-serving-cert" not found
Nov 03 12:18:34.602489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.602466    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db405b-a248-4f5d-8f00-15f9cd647c57-openshift-service-ca\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.604615 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.604597    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-metrics-certs\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:34.654785 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:34.654763    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56sb\" (UniqueName: \"kubernetes.io/projected/b7db405b-a248-4f5d-8f00-15f9cd647c57-kube-api-access-d56sb\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:35.144820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:35.138603    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:35.158214 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:35.158179    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-54zkf\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:35.372882 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:35.371949    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:35.810238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:35.810211    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"]
Nov 03 12:18:35.810989 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:18:35.810966    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7db405b_a248_4f5d_8f00_15f9cd647c57.slice/crio-f94258375d809cbe076598e4cbdadb4046513b07a5c55fbf960ca141340701da WatchSource:0}: Error finding container f94258375d809cbe076598e4cbdadb4046513b07a5c55fbf960ca141340701da: Status 404 returned error can't find the container with id f94258375d809cbe076598e4cbdadb4046513b07a5c55fbf960ca141340701da
Nov 03 12:18:36.513419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:36.513389    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf" event={"ID":"b7db405b-a248-4f5d-8f00-15f9cd647c57","Type":"ContainerStarted","Data":"f94258375d809cbe076598e4cbdadb4046513b07a5c55fbf960ca141340701da"}
Nov 03 12:18:37.521893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:37.521856    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf" event={"ID":"b7db405b-a248-4f5d-8f00-15f9cd647c57","Type":"ContainerStarted","Data":"d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89"}
Nov 03 12:18:37.522284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:37.521935    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:37.627844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:37.627792    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf" podStartSLOduration=2.662252889 podStartE2EDuration="3.627774522s" totalImagesPullingTime="965.521633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:37.627369326 +0000 UTC m=+1666.639836000" watchObservedRunningTime="2026-11-03 12:18:37.627774522 +0000 UTC m=+1666.640241185"
Nov 03 12:18:39.789840 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:39.789790    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:18:43.534867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:43.534838    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:18:46.639976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:46.639896    4909 generic.go:356] "Generic (PLEG): container finished" podID="7b3340a2-3347-4399-9535-4443015fc780" containerID="aa3f73bb30b8570f8dc9be50d5e58c11771291a6a8bdcdf114fe03af9d92707a" exitCode=0
Nov 03 12:18:46.640376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:46.639969    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" event={"ID":"7b3340a2-3347-4399-9535-4443015fc780","Type":"ContainerDied","Data":"aa3f73bb30b8570f8dc9be50d5e58c11771291a6a8bdcdf114fe03af9d92707a"}
Nov 03 12:18:46.640461 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:46.640444    4909 scope.go:122] "RemoveContainer" containerID="aa3f73bb30b8570f8dc9be50d5e58c11771291a6a8bdcdf114fe03af9d92707a"
Nov 03 12:18:46.642174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:46.642141    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:18:46.642277 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:46.642213    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:18:46.642332 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:46.642308    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:18:47.670070 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:47.670034    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" event={"ID":"7b3340a2-3347-4399-9535-4443015fc780","Type":"ContainerStarted","Data":"a447b178aceeb75eaf53ad05dc0f485324681d8cfc454f542f5f1899bd70a049"}
Nov 03 12:18:48.136779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:48.136750    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:18:48.148292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:48.148258    4909 patch_prober.go:29] interesting pod/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:18:48.148607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:48.148574    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 12:18:48.285981 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:48.285932    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"]
Nov 03 12:18:48.286244 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:48.286165    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" containerID="cri-o://d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57" gracePeriod=30
Nov 03 12:18:48.733168 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:48.733132    4909 patch_prober.go:29] interesting pod/e2e-api-b-931523-openshift-default-6b754b545c-czpt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:18:48.733555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:48.733191    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:18:49.137441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:49.137366    4909 patch_prober.go:29] interesting pod/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:18:49.137441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:49.137413    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 12:18:49.492922 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:49.492891    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:18:50.136800 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:50.136763    4909 patch_prober.go:29] interesting pod/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:18:50.137236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:50.136813    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 12:18:51.137128 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:51.137094    4909 patch_prober.go:29] interesting pod/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:18:51.137557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:51.137155    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 12:18:52.136828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:52.136794    4909 patch_prober.go:29] interesting pod/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:18:52.136996 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:52.136845    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 12:18:53.440457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.440426    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:18:53.442221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.442198    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:18:53.579443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.579417    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:18:53.699910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.699882    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-envoy\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700084 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.699996    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-token\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-token\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700084 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700037    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-socket\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700210 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700165    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-credential-socket\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700237    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-data\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700335 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700288    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-socket" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:53.700335 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700299    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-kube-api-access-znrxd\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-kube-api-access-znrxd\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700440 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700384    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-podinfo\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700440 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700430    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istiod-ca-cert\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700472    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-certs\") pod \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\" (UID: \"3e18d399-8a08-4e65-9310-b2cb6c7d1d63\") "
Nov 03 12:18:53.700552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700495    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-credential-socket" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:53.700742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700720    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-certs" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:53.700943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.700901    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-data" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:53.701269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.701221    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.701269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.701248    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.701269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.701262    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.701494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.701275    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.701494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.701361    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istiod-ca-cert" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:53.702899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.702875    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-envoy" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:53.703015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.702936    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-token" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:53.703592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.703571    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-podinfo" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:18:53.703681 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.703649    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-kube-api-access-znrxd" pod "3e18d399-8a08-4e65-9310-b2cb6c7d1d63" (UID: "3e18d399-8a08-4e65-9310-b2cb6c7d1d63"). InnerVolumeSpecName "kube-api-access-znrxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:53.803103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.803076    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.803103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.803101    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrxd\" (UniqueName: \"kubernetes.io/projected/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-kube-api-access-znrxd\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.803289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.803111    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.803289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.803121    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:53.803289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:53.803129    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e18d399-8a08-4e65-9310-b2cb6c7d1d63-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:54.028923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.028839    4909 generic.go:356] "Generic (PLEG): container finished" podID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerID="d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57" exitCode=0
Nov 03 12:18:54.028923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.028898    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"
Nov 03 12:18:54.029163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.028924    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" event={"ID":"3e18d399-8a08-4e65-9310-b2cb6c7d1d63","Type":"ContainerDied","Data":"d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57"}
Nov 03 12:18:54.029163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.029013    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9" event={"ID":"3e18d399-8a08-4e65-9310-b2cb6c7d1d63","Type":"ContainerDied","Data":"84be67745e9315113f01b26b34457626c69519ddd9255d2fdfb68ba2d30da1f9"}
Nov 03 12:18:54.029163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.029045    4909 scope.go:122] "RemoveContainer" containerID="d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57"
Nov 03 12:18:54.039986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.039970    4909 scope.go:122] "RemoveContainer" containerID="62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597"
Nov 03 12:18:54.048783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.048762    4909 scope.go:122] "RemoveContainer" containerID="d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57"
Nov 03 12:18:54.049034 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:54.049015    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57\": container with ID starting with d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57 not found: ID does not exist" containerID="d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57"
Nov 03 12:18:54.049114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.049046    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57"} err="failed to get container status \"d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57\": rpc error: code = NotFound desc = could not find container \"d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57\": container with ID starting with d91bd7e086528ec75e845551e36a22e6ecb44296c25e76bc6c28abb041e23f57 not found: ID does not exist"
Nov 03 12:18:54.049114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.049069    4909 scope.go:122] "RemoveContainer" containerID="62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597"
Nov 03 12:18:54.049316 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:18:54.049295    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597\": container with ID starting with 62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597 not found: ID does not exist" containerID="62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597"
Nov 03 12:18:54.049419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.049318    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597"} err="failed to get container status \"62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597\": rpc error: code = NotFound desc = could not find container \"62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597\": container with ID starting with 62abb2b1ed6a0d4f2b056551710ea341bf9ec05a86dfaa92d75e63a9be2bc597 not found: ID does not exist"
Nov 03 12:18:54.182148 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.182120    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"]
Nov 03 12:18:54.227921 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:54.227894    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-931523-openshift-default-6b754b545c-czpt9"]
Nov 03 12:18:55.124906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:55.124867    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" path="/var/lib/kubelet/pods/3e18d399-8a08-4e65-9310-b2cb6c7d1d63/volumes"
Nov 03 12:18:57.823689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:57.823650    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:18:58.530744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.530694    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"]
Nov 03 12:18:58.533112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.533076    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy"
Nov 03 12:18:58.533309 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.533285    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy"
Nov 03 12:18:58.533407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.533396    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy"
Nov 03 12:18:58.533481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.533471    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy"
Nov 03 12:18:58.533749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.533710    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy"
Nov 03 12:18:58.543637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.543242    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:58.649191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.647902    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1e151c\""
Nov 03 12:18:58.649191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.648129    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1e151c-dockercfg-th58v\""
Nov 03 12:18:58.708778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.685445    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5d408-a84e-4325-8fd2-486a724357d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:58.751569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.751530    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8z2d\" (UniqueName: \"kubernetes.io/projected/5eb5d408-a84e-4325-8fd2-486a724357d2-kube-api-access-s8z2d\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:58.803215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.803138    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"]
Nov 03 12:18:58.864483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.864441    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5d408-a84e-4325-8fd2-486a724357d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:58.864887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.864611    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8z2d\" (UniqueName: \"kubernetes.io/projected/5eb5d408-a84e-4325-8fd2-486a724357d2-kube-api-access-s8z2d\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:58.865264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.865246    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5d408-a84e-4325-8fd2-486a724357d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:58.982480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:58.975305    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8z2d\" (UniqueName: \"kubernetes.io/projected/5eb5d408-a84e-4325-8fd2-486a724357d2-kube-api-access-s8z2d\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:59.103750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:59.103647    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:18:59.502506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:18:59.502479    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"]
Nov 03 12:18:59.504274 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:18:59.504251    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb5d408_a84e_4325_8fd2_486a724357d2.slice/crio-43d9ce96e64bf88b10af42556ae4009777e239d45d2e8a8b17ef027baa1f1ac3 WatchSource:0}: Error finding container 43d9ce96e64bf88b10af42556ae4009777e239d45d2e8a8b17ef027baa1f1ac3: Status 404 returned error can't find the container with id 43d9ce96e64bf88b10af42556ae4009777e239d45d2e8a8b17ef027baa1f1ac3
Nov 03 12:19:00.154592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:00.154553    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h" event={"ID":"5eb5d408-a84e-4325-8fd2-486a724357d2","Type":"ContainerStarted","Data":"49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168"}
Nov 03 12:19:00.155199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:00.155170    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h" event={"ID":"5eb5d408-a84e-4325-8fd2-486a724357d2","Type":"ContainerStarted","Data":"43d9ce96e64bf88b10af42556ae4009777e239d45d2e8a8b17ef027baa1f1ac3"}
Nov 03 12:19:00.166102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:00.160382    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:19:00.268211 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:00.266927    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h" podStartSLOduration=2.266914157 podStartE2EDuration="2.266914157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:00.265645532 +0000 UTC m=+1689.278112196" watchObservedRunningTime="2026-11-03 12:19:00.266914157 +0000 UTC m=+1689.279380839"
Nov 03 12:19:07.383639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.383601    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"]
Nov 03 12:19:07.820398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.820368    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"]
Nov 03 12:19:07.821470 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.821447    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e18d399-8a08-4e65-9310-b2cb6c7d1d63" containerName="istio-proxy"
Nov 03 12:19:07.824619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.824596    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.882166 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.882138    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"]
Nov 03 12:19:07.888606 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.888583    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-45b1b7-openshift-default-dockercfg-49kk4\""
Nov 03 12:19:07.925058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925027    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-credential-socket\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925108    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-certs\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925167    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0642be-2076-4628-8206-686e571db79c-istio-podinfo\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925185    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm97f\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-kube-api-access-jm97f\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925214    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0642be-2076-4628-8206-686e571db79c-istiod-ca-cert\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925234    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-data\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925369    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-socket\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925388    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-envoy\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:07.925417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:07.925409    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-istio-token\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.033616 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.033568    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-certs\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.033942 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.033916    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0642be-2076-4628-8206-686e571db79c-istio-podinfo\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.034071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.034058    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm97f\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-kube-api-access-jm97f\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.034198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.034184    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0642be-2076-4628-8206-686e571db79c-istiod-ca-cert\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.034294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.034282    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-data\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.034500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.034486    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-socket\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.034583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.034572    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-envoy\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.034685 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.034674    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-istio-token\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.034847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.034833    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-credential-socket\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.035473 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.035447    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-credential-socket\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.035937 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.035915    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-certs\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.058490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.044512    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0642be-2076-4628-8206-686e571db79c-istiod-ca-cert\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.058676 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.044768    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-data\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.058836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.044987    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-socket\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.060267 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.060241    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0642be-2076-4628-8206-686e571db79c-istio-podinfo\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.076857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.063007    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-envoy\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.103628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.103597    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-istio-token\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.138256 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.131576    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm97f\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-kube-api-access-jm97f\") pod \"e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") " pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.138910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.138644    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:08.715738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.710981    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"]
Nov 03 12:19:08.761191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.758502    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:08.761191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.758599    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:08.761191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:08.758643    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:09.245425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:09.245390    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" event={"ID":"3a0642be-2076-4628-8206-686e571db79c","Type":"ContainerStarted","Data":"68bb5867fdb4a4bc87974129f554e000b7e575639e6e7aafbbd2c010b9925be1"}
Nov 03 12:19:09.245611 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:09.245445    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" event={"ID":"3a0642be-2076-4628-8206-686e571db79c","Type":"ContainerStarted","Data":"8321ea9b8bc78ad8a4c3178272a31bb992911a821ff2be656f92848f07d4386a"}
Nov 03 12:19:09.354902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:09.354852    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" podStartSLOduration=2.3548388940000002 podStartE2EDuration="2.354838894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:09.351946412 +0000 UTC m=+1698.364413075" watchObservedRunningTime="2026-11-03 12:19:09.354838894 +0000 UTC m=+1698.367305557"
Nov 03 12:19:09.629165 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:19:09.629075    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:19:10.169865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:10.169832    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:10.175287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:10.175262    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:10.253378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:10.253348    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:10.254215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:10.254197    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:19:12.177061 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.177026    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:19:12.348155 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.348126    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"]
Nov 03 12:19:12.352935 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.352911    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.384335 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.384297    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c" podUID="0ac230b1-3a43-428f-bad7-10795aa2bd7c" containerName="payload-pre-processing" containerID="cri-o://fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e" gracePeriod=25
Nov 03 12:19:12.385351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385201    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7j9\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-kube-api-access-hb7j9\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385247    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385278    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385373    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385417    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385465    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385659    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385724    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.385832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.385817    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.402913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.402886    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"]
Nov 03 12:19:12.487338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487307    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487350    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487385    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487503    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487546    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487664 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487623    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487666    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7j9\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-kube-api-access-hb7j9\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487677    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487759    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487799    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487803    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.487881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.487838    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.488099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.488030    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.488403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.488381    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.490025 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.489998    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.490275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.490257    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.541524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.541494    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.542777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.542745    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7j9\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-kube-api-access-hb7j9\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.633229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.633203    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:19:12.668316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.668288    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:12.690941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.690914    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac230b1-3a43-428f-bad7-10795aa2bd7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac230b1-3a43-428f-bad7-10795aa2bd7c-plugins-config-volume\") pod \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") "
Nov 03 12:19:12.691068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.691054    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac230b1-3a43-428f-bad7-10795aa2bd7c-kube-api-access-l95f8\" (UniqueName: \"kubernetes.io/projected/0ac230b1-3a43-428f-bad7-10795aa2bd7c-kube-api-access-l95f8\") pod \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\" (UID: \"0ac230b1-3a43-428f-bad7-10795aa2bd7c\") "
Nov 03 12:19:12.691438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.691414    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac230b1-3a43-428f-bad7-10795aa2bd7c-plugins-config-volume" pod "0ac230b1-3a43-428f-bad7-10795aa2bd7c" (UID: "0ac230b1-3a43-428f-bad7-10795aa2bd7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:12.691762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.691721    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac230b1-3a43-428f-bad7-10795aa2bd7c-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:12.695076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.694072    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac230b1-3a43-428f-bad7-10795aa2bd7c-kube-api-access-l95f8" pod "0ac230b1-3a43-428f-bad7-10795aa2bd7c" (UID: "0ac230b1-3a43-428f-bad7-10795aa2bd7c"). InnerVolumeSpecName "kube-api-access-l95f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:12.793439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:12.793406    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95f8\" (UniqueName: \"kubernetes.io/projected/0ac230b1-3a43-428f-bad7-10795aa2bd7c-kube-api-access-l95f8\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:13.095624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.095591    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"]
Nov 03 12:19:13.096295 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:19:13.096270    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f26cd9_898e_4ab3_97af_e4adccdd4e1d.slice/crio-13546aabaf9bbb38eaa692b3409cfbd2dbbb0dd5ad0ed6c7f7e44d5dc23ffe57 WatchSource:0}: Error finding container 13546aabaf9bbb38eaa692b3409cfbd2dbbb0dd5ad0ed6c7f7e44d5dc23ffe57: Status 404 returned error can't find the container with id 13546aabaf9bbb38eaa692b3409cfbd2dbbb0dd5ad0ed6c7f7e44d5dc23ffe57
Nov 03 12:19:13.098826 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.098771    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:13.098898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.098862    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:13.098898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.098893    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:13.276233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.276182    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" event={"ID":"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d","Type":"ContainerStarted","Data":"919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3"}
Nov 03 12:19:13.276710 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.276251    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" event={"ID":"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d","Type":"ContainerStarted","Data":"13546aabaf9bbb38eaa692b3409cfbd2dbbb0dd5ad0ed6c7f7e44d5dc23ffe57"}
Nov 03 12:19:13.277892 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.277863    4909 generic.go:356] "Generic (PLEG): container finished" podID="0ac230b1-3a43-428f-bad7-10795aa2bd7c" containerID="fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e" exitCode=0
Nov 03 12:19:13.278019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.277894    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c" event={"ID":"0ac230b1-3a43-428f-bad7-10795aa2bd7c","Type":"ContainerDied","Data":"fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e"}
Nov 03 12:19:13.278019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.277928    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"
Nov 03 12:19:13.278019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.277944    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c" event={"ID":"0ac230b1-3a43-428f-bad7-10795aa2bd7c","Type":"ContainerDied","Data":"1f596ae0af95816e8c14fe7e59715376f5dfda479d19e51e0d10c35d96398f73"}
Nov 03 12:19:13.278019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.277966    4909 scope.go:122] "RemoveContainer" containerID="fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e"
Nov 03 12:19:13.287769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.287749    4909 scope.go:122] "RemoveContainer" containerID="fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e"
Nov 03 12:19:13.288032 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:19:13.288010    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e\": container with ID starting with fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e not found: ID does not exist" containerID="fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e"
Nov 03 12:19:13.288120 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.288044    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e"} err="failed to get container status \"fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e\": rpc error: code = NotFound desc = could not find container \"fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e\": container with ID starting with fe5845c293768967c324ae0ad31447838ce858b860568333d9d295eec2eacf0e not found: ID does not exist"
Nov 03 12:19:13.387023 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.386931    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podStartSLOduration=1.38691869 podStartE2EDuration="1.38691869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:13.3853823 +0000 UTC m=+1702.397848962" watchObservedRunningTime="2026-11-03 12:19:13.38691869 +0000 UTC m=+1702.399385353"
Nov 03 12:19:13.505297 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.495762    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"]
Nov 03 12:19:13.514856 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.513645    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-86c57df8d7-ncl7c"]
Nov 03 12:19:13.677777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:13.672806    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:14.671436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:14.671398    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 12:19:14.671833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:14.671464    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 12:19:15.125559 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:15.125520    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac230b1-3a43-428f-bad7-10795aa2bd7c" path="/var/lib/kubelet/pods/0ac230b1-3a43-428f-bad7-10795aa2bd7c/volumes"
Nov 03 12:19:15.668889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:15.668857    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 12:19:15.669058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:15.668938    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 12:19:16.669124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:16.669090    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 12:19:16.669494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:16.669140    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 12:19:17.752724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:17.716332    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 12:19:17.752724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:17.716407    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 12:19:18.669293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:18.669249    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 12:19:18.669478 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:18.669326    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 12:19:19.668745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:19.668709    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 12:19:19.669246 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:19.668764    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 12:19:20.672802 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:20.672770    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:20.673195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:20.673172    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:20.673837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:20.673816    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:19:27.246040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.246006    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"]
Nov 03 12:19:27.246830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.246807    4909 cpu_manager.go:414] "containerMap: removing container" podUID="0ac230b1-3a43-428f-bad7-10795aa2bd7c" containerName="payload-pre-processing"
Nov 03 12:19:27.246935 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.246832    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac230b1-3a43-428f-bad7-10795aa2bd7c" containerName="payload-pre-processing"
Nov 03 12:19:27.246983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.246956    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac230b1-3a43-428f-bad7-10795aa2bd7c" containerName="payload-pre-processing"
Nov 03 12:19:27.251582 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.251562    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.349354 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.349324    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"]
Nov 03 12:19:27.397871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.397840    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.397891    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.397969    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.398030    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.398089    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zprx6\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-kube-api-access-zprx6\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398281 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.398243    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.398306    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398372 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.398345    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.398412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.398383    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982d09ad-629f-49a9-9d7b-78a949dd600e-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499260    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499313    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499351    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499394    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982d09ad-629f-49a9-9d7b-78a949dd600e-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499471    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499529    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499569    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499617    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499692    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499801    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zprx6\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-kube-api-access-zprx6\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.499970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499901    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.500027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.499999    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.500087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.500074    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.500244 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.500224    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982d09ad-629f-49a9-9d7b-78a949dd600e-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.502134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.502095    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.502287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.502269    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.552191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.552164    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.552357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.552338    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zprx6\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-kube-api-access-zprx6\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.567232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.567211    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:27.982556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.982524    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"]
Nov 03 12:19:27.982723 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:19:27.982561    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982d09ad_629f_49a9_9d7b_78a949dd600e.slice/crio-9bfb3dc425a2983fc41d7cbd236ec6a0cdb884dbd22c3e789363c16aa01e961e WatchSource:0}: Error finding container 9bfb3dc425a2983fc41d7cbd236ec6a0cdb884dbd22c3e789363c16aa01e961e: Status 404 returned error can't find the container with id 9bfb3dc425a2983fc41d7cbd236ec6a0cdb884dbd22c3e789363c16aa01e961e
Nov 03 12:19:27.985395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.985365    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:27.985509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.985429    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:27.985509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:27.985457    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:19:28.561721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:28.560980    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" event={"ID":"982d09ad-629f-49a9-9d7b-78a949dd600e","Type":"ContainerStarted","Data":"5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2"}
Nov 03 12:19:28.561721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:28.561033    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" event={"ID":"982d09ad-629f-49a9-9d7b-78a949dd600e","Type":"ContainerStarted","Data":"9bfb3dc425a2983fc41d7cbd236ec6a0cdb884dbd22c3e789363c16aa01e961e"}
Nov 03 12:19:28.568051 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:28.568023    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:28.674828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:28.674747    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" podStartSLOduration=1.6747225989999999 podStartE2EDuration="1.674722599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:28.670264449 +0000 UTC m=+1717.682731150" watchObservedRunningTime="2026-11-03 12:19:28.674722599 +0000 UTC m=+1717.687189266"
Nov 03 12:19:29.569530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:29.569497    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 12:19:29.569959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:29.569551    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 12:19:30.568241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:30.568202    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 12:19:30.568422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:30.568266    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 12:19:31.568054 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:31.568015    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 12:19:31.568443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:31.568068    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 12:19:32.569807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:32.569138    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 12:19:32.569807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:32.569200    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 12:19:33.568199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:33.568163    4909 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 12:19:33.568370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:33.568231    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 12:19:34.693322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:34.693290    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:34.694438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:34.694413    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:34.695263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:34.695220    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:19:39.564880 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:19:39.564839    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:19:47.843064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.843024    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"]
Nov 03 12:19:47.847573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.847556    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:47.942196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.942162    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1e151c-serving-cert\""
Nov 03 12:19:47.948196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.948170    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"]
Nov 03 12:19:47.951999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.951973    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:47.952165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.952142    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:47.952285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.952270    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-522c9\" (UniqueName: \"kubernetes.io/projected/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-kube-api-access-522c9\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:47.952345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:47.952315    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-metrics-certs\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.053801 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.053768    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.053984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.053876    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-522c9\" (UniqueName: \"kubernetes.io/projected/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-kube-api-access-522c9\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.053984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.053907    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-metrics-certs\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.053984 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:19:48.053944    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-serving-cert: secret "maas-api-e2e-ipp-a-1e151c-serving-cert" not found
Nov 03 12:19:48.054157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.054013    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.054157 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:19:48.054060    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls podName:5c18ee25-cbd5-446a-a2c7-c3487a838cc0 nodeName:}" failed. No retries permitted until 2026-11-03 12:19:48.5540356 +0000 UTC m=+1737.566502242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls") pod "maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" (UID: "5c18ee25-cbd5-446a-a2c7-c3487a838cc0") : secret "maas-api-e2e-ipp-a-1e151c-serving-cert" not found
Nov 03 12:19:48.054744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.054718    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.056576 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.056553    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-metrics-certs\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.133768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.130336    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-522c9\" (UniqueName: \"kubernetes.io/projected/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-kube-api-access-522c9\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.569186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.569150    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.571504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.571476    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:48.761594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:48.761567    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:49.131819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:49.131791    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"]
Nov 03 12:19:49.132229 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:19:49.132159    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c18ee25_cbd5_446a_a2c7_c3487a838cc0.slice/crio-b8b603fce0ce42e2c66fa0fe6a8b8ef786c63ff9c978109747dcf218666bf0dd WatchSource:0}: Error finding container b8b603fce0ce42e2c66fa0fe6a8b8ef786c63ff9c978109747dcf218666bf0dd: Status 404 returned error can't find the container with id b8b603fce0ce42e2c66fa0fe6a8b8ef786c63ff9c978109747dcf218666bf0dd
Nov 03 12:19:49.133552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:49.133531    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:19:49.777437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:49.777398    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" event={"ID":"5c18ee25-cbd5-446a-a2c7-c3487a838cc0","Type":"ContainerStarted","Data":"b8b603fce0ce42e2c66fa0fe6a8b8ef786c63ff9c978109747dcf218666bf0dd"}
Nov 03 12:19:50.814953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:50.814913    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" event={"ID":"5c18ee25-cbd5-446a-a2c7-c3487a838cc0","Type":"ContainerStarted","Data":"083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc"}
Nov 03 12:19:50.816388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:50.816362    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:19:50.922554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:50.922506    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" podStartSLOduration=3.0384342540000002 podStartE2EDuration="3.922492615s" totalImagesPullingTime="884.058361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:50.920778836 +0000 UTC m=+1739.933245499" watchObservedRunningTime="2026-11-03 12:19:50.922492615 +0000 UTC m=+1739.934959317"
Nov 03 12:19:55.969943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:55.969911    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"]
Nov 03 12:19:55.970376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:55.970097    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy" containerID="cri-o://0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808" gracePeriod=30
Nov 03 12:19:56.311005 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:56.310906    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:19:57.859322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:19:57.859288    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:20:01.249104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.248999    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:20:01.279964 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.279932    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-envoy\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280075    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-data\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280136    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-credential-socket\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280206    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-socket\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280244    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-kube-api-access-kd27w\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-kube-api-access-kd27w\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280288    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-certs\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280339    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-podinfo\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280382    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-data" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.280443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280397    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-token\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-token\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280515    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995db966-94b9-4691-a0ec-c5e88b54d7bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995db966-94b9-4691-a0ec-c5e88b54d7bb-istiod-ca-cert\") pod \"995db966-94b9-4691-a0ec-c5e88b54d7bb\" (UID: \"995db966-94b9-4691-a0ec-c5e88b54d7bb\") "
Nov 03 12:20:01.280684 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280640    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-certs" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.280802 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.280718    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-socket" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.281076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.281045    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995db966-94b9-4691-a0ec-c5e88b54d7bb-istiod-ca-cert" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:01.281232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.281211    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.281320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.281238    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.281320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.281254    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.281563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.281530    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-credential-socket" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.282401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.282381    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-envoy" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.282861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.282840    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-podinfo" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:20:01.283085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.283061    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-token" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:01.283302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.283274    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-kube-api-access-kd27w" pod "995db966-94b9-4691-a0ec-c5e88b54d7bb" (UID: "995db966-94b9-4691-a0ec-c5e88b54d7bb"). InnerVolumeSpecName "kube-api-access-kd27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:01.382295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.382263    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.382295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.382291    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd27w\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-kube-api-access-kd27w\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.382295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.382303    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.382590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.382313    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.382590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.382326    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995db966-94b9-4691-a0ec-c5e88b54d7bb-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.382590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.382335    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995db966-94b9-4691-a0ec-c5e88b54d7bb-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.847847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.847818    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"]
Nov 03 12:20:01.906051 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.906025    4909 generic.go:356] "Generic (PLEG): container finished" podID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerID="0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808" exitCode=0
Nov 03 12:20:01.906197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.906064    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" event={"ID":"995db966-94b9-4691-a0ec-c5e88b54d7bb","Type":"ContainerDied","Data":"0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808"}
Nov 03 12:20:01.906197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.906068    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"
Nov 03 12:20:01.906197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.906101    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499" event={"ID":"995db966-94b9-4691-a0ec-c5e88b54d7bb","Type":"ContainerDied","Data":"8a80ccdf5eca47e0321ca161094f590cb3d9f623dc2446cd4c8bb121a18c8989"}
Nov 03 12:20:01.906197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.906122    4909 scope.go:122] "RemoveContainer" containerID="0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808"
Nov 03 12:20:01.916789 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.916759    4909 scope.go:122] "RemoveContainer" containerID="04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b"
Nov 03 12:20:01.925405 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.925384    4909 scope.go:122] "RemoveContainer" containerID="0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808"
Nov 03 12:20:01.925676 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:20:01.925659    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808\": container with ID starting with 0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808 not found: ID does not exist" containerID="0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808"
Nov 03 12:20:01.925757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.925685    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808"} err="failed to get container status \"0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808\": rpc error: code = NotFound desc = could not find container \"0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808\": container with ID starting with 0dc1186282311888fffe8cf8f7e1c1569454d4b6515159dd45d731f498b2d808 not found: ID does not exist"
Nov 03 12:20:01.925757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.925728    4909 scope.go:122] "RemoveContainer" containerID="04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b"
Nov 03 12:20:01.925976 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:20:01.925959    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b\": container with ID starting with 04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b not found: ID does not exist" containerID="04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b"
Nov 03 12:20:01.926013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.925982    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b"} err="failed to get container status \"04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b\": rpc error: code = NotFound desc = could not find container \"04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b\": container with ID starting with 04137a96557b8f2908d992e1190a93698cf05e11c3f7e73f4b91f24e02c2a07b not found: ID does not exist"
Nov 03 12:20:01.946310 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:01.946291    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"]
Nov 03 12:20:02.167122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:02.167021    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"]
Nov 03 12:20:02.170276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:02.170251    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f6c034-openshift-default-5bdff5854f-5b499"]
Nov 03 12:20:03.124440 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:03.124411    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" path="/var/lib/kubelet/pods/995db966-94b9-4691-a0ec-c5e88b54d7bb/volumes"
Nov 03 12:20:06.848727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:06.848668    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb" podUID="73188b29-c22c-413f-8569-1cc7e0c738e6" containerName="payload-processing" containerID="cri-o://45b9426da44fca678967419616cb6482bf257e9055bccd4446ae9128ed212992" gracePeriod=25
Nov 03 12:20:06.947478 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:06.947446    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4" podUID="60313337-ed50-4185-9a57-3e162f277efb" containerName="payload-pre-processing" containerID="cri-o://2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905" gracePeriod=25
Nov 03 12:20:06.977456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:06.977433    4909 generic.go:356] "Generic (PLEG): container finished" podID="73188b29-c22c-413f-8569-1cc7e0c738e6" containerID="45b9426da44fca678967419616cb6482bf257e9055bccd4446ae9128ed212992" exitCode=0
Nov 03 12:20:06.977558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:06.977525    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb" event={"ID":"73188b29-c22c-413f-8569-1cc7e0c738e6","Type":"ContainerDied","Data":"45b9426da44fca678967419616cb6482bf257e9055bccd4446ae9128ed212992"}
Nov 03 12:20:07.131065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.131040    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:20:07.189278 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.189249    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73188b29-c22c-413f-8569-1cc7e0c738e6-kube-api-access-bcdvw\" (UniqueName: \"kubernetes.io/projected/73188b29-c22c-413f-8569-1cc7e0c738e6-kube-api-access-bcdvw\") pod \"73188b29-c22c-413f-8569-1cc7e0c738e6\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") "
Nov 03 12:20:07.189460 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.189444    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73188b29-c22c-413f-8569-1cc7e0c738e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73188b29-c22c-413f-8569-1cc7e0c738e6-plugins-config-volume\") pod \"73188b29-c22c-413f-8569-1cc7e0c738e6\" (UID: \"73188b29-c22c-413f-8569-1cc7e0c738e6\") "
Nov 03 12:20:07.190529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.190463    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73188b29-c22c-413f-8569-1cc7e0c738e6-plugins-config-volume" pod "73188b29-c22c-413f-8569-1cc7e0c738e6" (UID: "73188b29-c22c-413f-8569-1cc7e0c738e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:07.192251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.192216    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73188b29-c22c-413f-8569-1cc7e0c738e6-kube-api-access-bcdvw" pod "73188b29-c22c-413f-8569-1cc7e0c738e6" (UID: "73188b29-c22c-413f-8569-1cc7e0c738e6"). InnerVolumeSpecName "kube-api-access-bcdvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:07.232203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.232176    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:20:07.298043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.297930    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60313337-ed50-4185-9a57-3e162f277efb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60313337-ed50-4185-9a57-3e162f277efb-plugins-config-volume\") pod \"60313337-ed50-4185-9a57-3e162f277efb\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") "
Nov 03 12:20:07.298221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.298149    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60313337-ed50-4185-9a57-3e162f277efb-kube-api-access-94j8k\" (UniqueName: \"kubernetes.io/projected/60313337-ed50-4185-9a57-3e162f277efb-kube-api-access-94j8k\") pod \"60313337-ed50-4185-9a57-3e162f277efb\" (UID: \"60313337-ed50-4185-9a57-3e162f277efb\") "
Nov 03 12:20:07.298765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.298737    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60313337-ed50-4185-9a57-3e162f277efb-plugins-config-volume" pod "60313337-ed50-4185-9a57-3e162f277efb" (UID: "60313337-ed50-4185-9a57-3e162f277efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:07.298924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.298757    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73188b29-c22c-413f-8569-1cc7e0c738e6-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:07.299050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.299034    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcdvw\" (UniqueName: \"kubernetes.io/projected/73188b29-c22c-413f-8569-1cc7e0c738e6-kube-api-access-bcdvw\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:07.310726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.301756    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60313337-ed50-4185-9a57-3e162f277efb-kube-api-access-94j8k" pod "60313337-ed50-4185-9a57-3e162f277efb" (UID: "60313337-ed50-4185-9a57-3e162f277efb"). InnerVolumeSpecName "kube-api-access-94j8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:07.400598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.400520    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60313337-ed50-4185-9a57-3e162f277efb-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:07.400598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.400550    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94j8k\" (UniqueName: \"kubernetes.io/projected/60313337-ed50-4185-9a57-3e162f277efb-kube-api-access-94j8k\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:07.985551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.985521    4909 generic.go:356] "Generic (PLEG): container finished" podID="60313337-ed50-4185-9a57-3e162f277efb" containerID="2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905" exitCode=0
Nov 03 12:20:07.986030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.985566    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"
Nov 03 12:20:07.986030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.985604    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4" event={"ID":"60313337-ed50-4185-9a57-3e162f277efb","Type":"ContainerDied","Data":"2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905"}
Nov 03 12:20:07.986030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.985657    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4" event={"ID":"60313337-ed50-4185-9a57-3e162f277efb","Type":"ContainerDied","Data":"c7051baf7a875f86962de89d9fd9d641d77dcd3e67d10812740bd9da85b3937a"}
Nov 03 12:20:07.986030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.985680    4909 scope.go:122] "RemoveContainer" containerID="2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905"
Nov 03 12:20:07.987219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.987198    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb" event={"ID":"73188b29-c22c-413f-8569-1cc7e0c738e6","Type":"ContainerDied","Data":"334056a51056b43f9a510408d8969d2969edf370df360a03b0c5dabf1a096207"}
Nov 03 12:20:07.987329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.987254    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"
Nov 03 12:20:07.996312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.996273    4909 scope.go:122] "RemoveContainer" containerID="2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905"
Nov 03 12:20:07.997779 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:20:07.997746    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905\": container with ID starting with 2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905 not found: ID does not exist" containerID="2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905"
Nov 03 12:20:07.997891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.997790    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905"} err="failed to get container status \"2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905\": rpc error: code = NotFound desc = could not find container \"2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905\": container with ID starting with 2581e6bebaf9a79cae7a6a3949cc8ccac4cf6375187b24ee188ab8b9b0c9d905 not found: ID does not exist"
Nov 03 12:20:07.997891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:07.997818    4909 scope.go:122] "RemoveContainer" containerID="45b9426da44fca678967419616cb6482bf257e9055bccd4446ae9128ed212992"
Nov 03 12:20:08.140821 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:08.140743    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"]
Nov 03 12:20:08.186044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:08.186019    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89a88d-7c5b649d8f-l2khb"]
Nov 03 12:20:08.286769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:08.286691    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"]
Nov 03 12:20:08.289792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:08.289770    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89a88d-5dcf4b8d9f-6swh4"]
Nov 03 12:20:09.123913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:09.123883    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60313337-ed50-4185-9a57-3e162f277efb" path="/var/lib/kubelet/pods/60313337-ed50-4185-9a57-3e162f277efb/volumes"
Nov 03 12:20:09.124307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:09.124293    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73188b29-c22c-413f-8569-1cc7e0c738e6" path="/var/lib/kubelet/pods/73188b29-c22c-413f-8569-1cc7e0c738e6/volumes"
Nov 03 12:20:09.596396 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:20:09.596360    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:20:09.931393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:09.931313    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"]
Nov 03 12:20:09.931618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:09.931587    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy" containerID="cri-o://a447b178aceeb75eaf53ad05dc0f485324681d8cfc454f542f5f1899bd70a049" gracePeriod=30
Nov 03 12:20:15.126826 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.126790    4909 generic.go:356] "Generic (PLEG): container finished" podID="7b3340a2-3347-4399-9535-4443015fc780" containerID="a447b178aceeb75eaf53ad05dc0f485324681d8cfc454f542f5f1899bd70a049" exitCode=0
Nov 03 12:20:15.128987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.128949    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" event={"ID":"7b3340a2-3347-4399-9535-4443015fc780","Type":"ContainerDied","Data":"a447b178aceeb75eaf53ad05dc0f485324681d8cfc454f542f5f1899bd70a049"}
Nov 03 12:20:15.129119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.129025    4909 scope.go:122] "RemoveContainer" containerID="aa3f73bb30b8570f8dc9be50d5e58c11771291a6a8bdcdf114fe03af9d92707a"
Nov 03 12:20:15.204220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.204165    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:20:15.307395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307359    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-certs\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.307558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307403    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-data\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.307558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307421    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-socket\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.307558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307439    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-istio-token\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-istio-token\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.307558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307490    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-envoy\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.307772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307579    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-kube-api-access-s4c5d\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-kube-api-access-s4c5d\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.307772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307650    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-certs" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:15.307772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307670    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-data" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:15.307772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307686    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-socket" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:15.307772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307742    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3340a2-3347-4399-9535-4443015fc780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3340a2-3347-4399-9535-4443015fc780-istiod-ca-cert\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.308037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307792    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b3340a2-3347-4399-9535-4443015fc780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3340a2-3347-4399-9535-4443015fc780-istio-podinfo\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.308037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.307915    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-credential-socket\") pod \"7b3340a2-3347-4399-9535-4443015fc780\" (UID: \"7b3340a2-3347-4399-9535-4443015fc780\") "
Nov 03 12:20:15.308144 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.308101    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3340a2-3347-4399-9535-4443015fc780-istiod-ca-cert" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:15.308344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.308309    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-credential-socket" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:15.308522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.308501    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.308599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.308525    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.308599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.308540    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.308599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.308549    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.308599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.308558    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3340a2-3347-4399-9535-4443015fc780-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.309925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.309902    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-kube-api-access-s4c5d" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "kube-api-access-s4c5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:15.310178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.310155    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-istio-token" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:15.310278 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.310208    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-envoy" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:15.310278 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.310225    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b3340a2-3347-4399-9535-4443015fc780-istio-podinfo" pod "7b3340a2-3347-4399-9535-4443015fc780" (UID: "7b3340a2-3347-4399-9535-4443015fc780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:20:15.409476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.409445    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3340a2-3347-4399-9535-4443015fc780-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.409476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.409479    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.409661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.409494    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3340a2-3347-4399-9535-4443015fc780-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:15.409661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:15.409505    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4c5d\" (UniqueName: \"kubernetes.io/projected/7b3340a2-3347-4399-9535-4443015fc780-kube-api-access-s4c5d\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:16.133805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:16.133778    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"
Nov 03 12:20:16.134214 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:16.133815    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn" event={"ID":"7b3340a2-3347-4399-9535-4443015fc780","Type":"ContainerDied","Data":"dde007e0a1e5ce576a487b964bcbfdc425aa9d406001cfd9cac481f19159a28f"}
Nov 03 12:20:16.134214 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:16.133870    4909 scope.go:122] "RemoveContainer" containerID="a447b178aceeb75eaf53ad05dc0f485324681d8cfc454f542f5f1899bd70a049"
Nov 03 12:20:16.333018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:16.332984    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"]
Nov 03 12:20:16.378588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:16.378556    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-89a88d-openshift-default-58886f875f-s4bbn"]
Nov 03 12:20:17.124173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:17.124144    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3340a2-3347-4399-9535-4443015fc780" path="/var/lib/kubelet/pods/7b3340a2-3347-4399-9535-4443015fc780/volumes"
Nov 03 12:20:23.449533 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:23.449502    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:20:24.026654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:24.026620    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"]
Nov 03 12:20:29.027758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.027721    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb" podUID="dd899088-3994-4cdc-bca3-97bd4cda35e9" containerName="payload-processing" containerID="cri-o://af0bb0f669c7a12bb44b48705c8e2f38173632963811650ab71437c3f23c820e" gracePeriod=25
Nov 03 12:20:29.222443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.222405    4909 generic.go:356] "Generic (PLEG): container finished" podID="dd899088-3994-4cdc-bca3-97bd4cda35e9" containerID="af0bb0f669c7a12bb44b48705c8e2f38173632963811650ab71437c3f23c820e" exitCode=0
Nov 03 12:20:29.222609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.222513    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb" event={"ID":"dd899088-3994-4cdc-bca3-97bd4cda35e9","Type":"ContainerDied","Data":"af0bb0f669c7a12bb44b48705c8e2f38173632963811650ab71437c3f23c820e"}
Nov 03 12:20:29.313983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.313959    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:20:29.497714 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.497670    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd899088-3994-4cdc-bca3-97bd4cda35e9-kube-api-access-hzpgl\" (UniqueName: \"kubernetes.io/projected/dd899088-3994-4cdc-bca3-97bd4cda35e9-kube-api-access-hzpgl\") pod \"dd899088-3994-4cdc-bca3-97bd4cda35e9\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") "
Nov 03 12:20:29.497983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.497965    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd899088-3994-4cdc-bca3-97bd4cda35e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd899088-3994-4cdc-bca3-97bd4cda35e9-plugins-config-volume\") pod \"dd899088-3994-4cdc-bca3-97bd4cda35e9\" (UID: \"dd899088-3994-4cdc-bca3-97bd4cda35e9\") "
Nov 03 12:20:29.498314 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.498287    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd899088-3994-4cdc-bca3-97bd4cda35e9-plugins-config-volume" pod "dd899088-3994-4cdc-bca3-97bd4cda35e9" (UID: "dd899088-3994-4cdc-bca3-97bd4cda35e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:29.499842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.499816    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd899088-3994-4cdc-bca3-97bd4cda35e9-kube-api-access-hzpgl" pod "dd899088-3994-4cdc-bca3-97bd4cda35e9" (UID: "dd899088-3994-4cdc-bca3-97bd4cda35e9"). InnerVolumeSpecName "kube-api-access-hzpgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:29.599577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.599498    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd899088-3994-4cdc-bca3-97bd4cda35e9-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:29.599577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:29.599532    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzpgl\" (UniqueName: \"kubernetes.io/projected/dd899088-3994-4cdc-bca3-97bd4cda35e9-kube-api-access-hzpgl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:30.230649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:30.230611    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb" event={"ID":"dd899088-3994-4cdc-bca3-97bd4cda35e9","Type":"ContainerDied","Data":"fbda50ca79d165b643ba2ff9aad79cc3f9d3c27e9da2190456eeacc2ba05b01e"}
Nov 03 12:20:30.231126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:30.230680    4909 scope.go:122] "RemoveContainer" containerID="af0bb0f669c7a12bb44b48705c8e2f38173632963811650ab71437c3f23c820e"
Nov 03 12:20:30.231126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:30.230634    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"
Nov 03 12:20:30.419793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:30.416756    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"]
Nov 03 12:20:30.438827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:30.437382    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69fd8043-58c77c8499-cgmbb"]
Nov 03 12:20:31.124426 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:31.124397    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd899088-3994-4cdc-bca3-97bd4cda35e9" path="/var/lib/kubelet/pods/dd899088-3994-4cdc-bca3-97bd4cda35e9/volumes"
Nov 03 12:20:39.564345 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:20:39.564311    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:20:49.433137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:49.433104    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:20:51.343999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:51.343969    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:20:51.345477 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:51.345455    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:20:53.687189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:53.687153    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"]
Nov 03 12:20:58.688196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:58.688126    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h" podUID="5eb5d408-a84e-4325-8fd2-486a724357d2" containerName="payload-pre-processing" containerID="cri-o://49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168" gracePeriod=25
Nov 03 12:20:58.993484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:58.993459    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:20:59.114764 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.114733    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb5d408-a84e-4325-8fd2-486a724357d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5d408-a84e-4325-8fd2-486a724357d2-plugins-config-volume\") pod \"5eb5d408-a84e-4325-8fd2-486a724357d2\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") "
Nov 03 12:20:59.114983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.114891    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb5d408-a84e-4325-8fd2-486a724357d2-kube-api-access-s8z2d\" (UniqueName: \"kubernetes.io/projected/5eb5d408-a84e-4325-8fd2-486a724357d2-kube-api-access-s8z2d\") pod \"5eb5d408-a84e-4325-8fd2-486a724357d2\" (UID: \"5eb5d408-a84e-4325-8fd2-486a724357d2\") "
Nov 03 12:20:59.116720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.115647    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb5d408-a84e-4325-8fd2-486a724357d2-plugins-config-volume" pod "5eb5d408-a84e-4325-8fd2-486a724357d2" (UID: "5eb5d408-a84e-4325-8fd2-486a724357d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:59.126938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.126908    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5d408-a84e-4325-8fd2-486a724357d2-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:59.140906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.140860    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb5d408-a84e-4325-8fd2-486a724357d2-kube-api-access-s8z2d" pod "5eb5d408-a84e-4325-8fd2-486a724357d2" (UID: "5eb5d408-a84e-4325-8fd2-486a724357d2"). InnerVolumeSpecName "kube-api-access-s8z2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:59.228089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.228003    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8z2d\" (UniqueName: \"kubernetes.io/projected/5eb5d408-a84e-4325-8fd2-486a724357d2-kube-api-access-s8z2d\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:59.497323 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.497244    4909 generic.go:356] "Generic (PLEG): container finished" podID="5eb5d408-a84e-4325-8fd2-486a724357d2" containerID="49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168" exitCode=0
Nov 03 12:20:59.497323 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.497307    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"
Nov 03 12:20:59.497517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.497320    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h" event={"ID":"5eb5d408-a84e-4325-8fd2-486a724357d2","Type":"ContainerDied","Data":"49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168"}
Nov 03 12:20:59.497517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.497371    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h" event={"ID":"5eb5d408-a84e-4325-8fd2-486a724357d2","Type":"ContainerDied","Data":"43d9ce96e64bf88b10af42556ae4009777e239d45d2e8a8b17ef027baa1f1ac3"}
Nov 03 12:20:59.497517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.497392    4909 scope.go:122] "RemoveContainer" containerID="49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168"
Nov 03 12:20:59.507891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.507869    4909 scope.go:122] "RemoveContainer" containerID="49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168"
Nov 03 12:20:59.508150 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:20:59.508131    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168\": container with ID starting with 49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168 not found: ID does not exist" containerID="49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168"
Nov 03 12:20:59.508192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.508160    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168"} err="failed to get container status \"49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168\": rpc error: code = NotFound desc = could not find container \"49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168\": container with ID starting with 49245fb59186c275d5fe0321cb5b83986defeab05aab1890ce2ba179d9539168 not found: ID does not exist"
Nov 03 12:20:59.650566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.650533    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"]
Nov 03 12:20:59.696406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:20:59.696370    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-78c7c6bd55-g976h"]
Nov 03 12:21:01.124972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:01.124941    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb5d408-a84e-4325-8fd2-486a724357d2" path="/var/lib/kubelet/pods/5eb5d408-a84e-4325-8fd2-486a724357d2/volumes"
Nov 03 12:21:09.588062 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:21:09.588029    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:21:10.780423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.780389    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"]
Nov 03 12:21:10.781059 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781037    4909 cpu_manager.go:414] "containerMap: removing container" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy"
Nov 03 12:21:10.781059 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781056    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy"
Nov 03 12:21:10.781059 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781066    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781072    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781100    4909 cpu_manager.go:414] "containerMap: removing container" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781106    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781116    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5eb5d408-a84e-4325-8fd2-486a724357d2" containerName="payload-pre-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781121    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb5d408-a84e-4325-8fd2-486a724357d2" containerName="payload-pre-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781131    4909 cpu_manager.go:414] "containerMap: removing container" podUID="dd899088-3994-4cdc-bca3-97bd4cda35e9" containerName="payload-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781136    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd899088-3994-4cdc-bca3-97bd4cda35e9" containerName="payload-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781146    4909 cpu_manager.go:414] "containerMap: removing container" podUID="73188b29-c22c-413f-8569-1cc7e0c738e6" containerName="payload-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781151    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73188b29-c22c-413f-8569-1cc7e0c738e6" containerName="payload-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781170    4909 cpu_manager.go:414] "containerMap: removing container" podUID="60313337-ed50-4185-9a57-3e162f277efb" containerName="payload-pre-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781175    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60313337-ed50-4185-9a57-3e162f277efb" containerName="payload-pre-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781265    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781273    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb5d408-a84e-4325-8fd2-486a724357d2" containerName="payload-pre-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781281    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd899088-3994-4cdc-bca3-97bd4cda35e9" containerName="payload-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781287    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781292    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="73188b29-c22c-413f-8569-1cc7e0c738e6" containerName="payload-processing"
Nov 03 12:21:10.781303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.781301    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="60313337-ed50-4185-9a57-3e162f277efb" containerName="payload-pre-processing"
Nov 03 12:21:10.786460 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.786441    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.855988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.855956    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"]
Nov 03 12:21:10.874660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.874621    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-metrics-certs\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.874854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.874808    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8kq\" (UniqueName: \"kubernetes.io/projected/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-kube-api-access-vd8kq\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.874945 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.874930    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.875007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.874987    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.880665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.880638    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2f3eb2-serving-cert\""
Nov 03 12:21:10.976768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.976729    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.976933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.976796    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.976933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.976924    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-metrics-certs\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.977013 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:21:10.976963    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-serving-cert: secret "maas-api-e2e-ipp-a-2f3eb2-serving-cert" not found
Nov 03 12:21:10.977013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.976990    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8kq\" (UniqueName: \"kubernetes.io/projected/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-kube-api-access-vd8kq\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.977091 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:21:10.977034    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls podName:27c8a7f1-2405-44d9-a7dc-38ce7e68c122 nodeName:}" failed. No retries permitted until 2026-11-03 12:21:11.477013096 +0000 UTC m=+1820.489479746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls") pod "maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48" (UID: "27c8a7f1-2405-44d9-a7dc-38ce7e68c122") : secret "maas-api-e2e-ipp-a-2f3eb2-serving-cert" not found
Nov 03 12:21:10.977402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.977385    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:10.979384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:10.979365    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-metrics-certs\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:11.124906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:11.115418    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8kq\" (UniqueName: \"kubernetes.io/projected/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-kube-api-access-vd8kq\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:11.572722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:11.496930    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:11.572722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:11.519239    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:11.700776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:11.700749    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:12.177909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:12.177883    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"]
Nov 03 12:21:12.179382 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:21:12.179354    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c8a7f1_2405_44d9_a7dc_38ce7e68c122.slice/crio-b8dfb832436ddcf51e19a3f3963f7c496a3d799afaf5b0ef5bf713a384c34d42 WatchSource:0}: Error finding container b8dfb832436ddcf51e19a3f3963f7c496a3d799afaf5b0ef5bf713a384c34d42: Status 404 returned error can't find the container with id b8dfb832436ddcf51e19a3f3963f7c496a3d799afaf5b0ef5bf713a384c34d42
Nov 03 12:21:12.635117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:12.635076    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48" event={"ID":"27c8a7f1-2405-44d9-a7dc-38ce7e68c122","Type":"ContainerStarted","Data":"b8dfb832436ddcf51e19a3f3963f7c496a3d799afaf5b0ef5bf713a384c34d42"}
Nov 03 12:21:13.644599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:13.644558    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48" event={"ID":"27c8a7f1-2405-44d9-a7dc-38ce7e68c122","Type":"ContainerStarted","Data":"0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb"}
Nov 03 12:21:13.644986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:13.644683    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:13.860044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:13.859135    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48" podStartSLOduration=3.017601641 podStartE2EDuration="3.859111641s" totalImagesPullingTime="841.51ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:13.755155466 +0000 UTC m=+1822.767622145" watchObservedRunningTime="2026-11-03 12:21:13.859111641 +0000 UTC m=+1822.871578305"
Nov 03 12:21:15.659283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:15.659250    4909 generic.go:356] "Generic (PLEG): container finished" podID="3a0642be-2076-4628-8206-686e571db79c" containerID="68bb5867fdb4a4bc87974129f554e000b7e575639e6e7aafbbd2c010b9925be1" exitCode=0
Nov 03 12:21:15.659836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:15.659332    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" event={"ID":"3a0642be-2076-4628-8206-686e571db79c","Type":"ContainerDied","Data":"68bb5867fdb4a4bc87974129f554e000b7e575639e6e7aafbbd2c010b9925be1"}
Nov 03 12:21:15.659909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:15.659892    4909 scope.go:122] "RemoveContainer" containerID="68bb5867fdb4a4bc87974129f554e000b7e575639e6e7aafbbd2c010b9925be1"
Nov 03 12:21:15.661409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:15.661379    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:21:15.661493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:15.661465    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:21:15.661535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:15.661498    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:21:16.671261 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:16.671218    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" event={"ID":"3a0642be-2076-4628-8206-686e571db79c","Type":"ContainerStarted","Data":"b05ed1cbec0304bc1b912f17a9f9dca4469d27012788a6b175fb956cebe4aff1"}
Nov 03 12:21:17.169856 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:17.169816    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:21:17.170051 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:17.170025    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 12:21:17.170150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:17.170082    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 12:21:18.170225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:18.170191    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 12:21:18.170642 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:18.170262    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 12:21:19.169677 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:19.169645    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 12:21:19.169908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:19.169716    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 12:21:19.666783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:19.666748    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:21:20.169343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:20.169306    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 12:21:20.169522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:20.169360    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 12:21:21.170107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:21.170074    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 12:21:21.170505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:21.170137    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 12:21:22.173398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:22.173368    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:21:22.173883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:22.173676    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:21:22.174438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:22.174419    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:21:30.379911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:30.379875    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:21:39.565248 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:21:39.565169    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:21:52.371290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.371259    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"]
Nov 03 12:21:52.372143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.372117    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy"
Nov 03 12:21:52.372143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.372142    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy"
Nov 03 12:21:52.372343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.372308    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="995db966-94b9-4691-a0ec-c5e88b54d7bb" containerName="istio-proxy"
Nov 03 12:21:52.372343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.372324    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3340a2-3347-4399-9535-4443015fc780" containerName="istio-proxy"
Nov 03 12:21:52.377359 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.377342    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.429125 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.429098    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"]
Nov 03 12:21:52.470797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.470751    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-583bcf-serving-cert\""
Nov 03 12:21:52.523734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.523686    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f7294-7021-48a8-a979-e447011bc9c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.524062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.524032    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.524245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.524227    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-metrics-certs\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.524402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.524380    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2sv\" (UniqueName: \"kubernetes.io/projected/1d6f7294-7021-48a8-a979-e447011bc9c1-kube-api-access-nd2sv\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.636957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.636863    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-metrics-certs\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.638019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.637988    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2sv\" (UniqueName: \"kubernetes.io/projected/1d6f7294-7021-48a8-a979-e447011bc9c1-kube-api-access-nd2sv\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.638603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.638581    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f7294-7021-48a8-a979-e447011bc9c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.639635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.639512    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f7294-7021-48a8-a979-e447011bc9c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.662830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.662795    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.663393 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:21:52.663203    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-serving-cert: secret "maas-api-e2e-ipp-b-583bcf-serving-cert" not found
Nov 03 12:21:52.663600 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:21:52.663587    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls podName:1d6f7294-7021-48a8-a979-e447011bc9c1 nodeName:}" failed. No retries permitted until 2026-11-03 12:21:53.163563939 +0000 UTC m=+1862.176030595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls") pod "maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7" (UID: "1d6f7294-7021-48a8-a979-e447011bc9c1") : secret "maas-api-e2e-ipp-b-583bcf-serving-cert" not found
Nov 03 12:21:52.691270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.691224    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-metrics-certs\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:52.711536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:52.711502    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2sv\" (UniqueName: \"kubernetes.io/projected/1d6f7294-7021-48a8-a979-e447011bc9c1-kube-api-access-nd2sv\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:53.170222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:53.170186    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:53.172641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:53.172620    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:53.291596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:53.291569    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:53.660265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:53.660236    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"]
Nov 03 12:21:53.661239 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:21:53.661212    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6f7294_7021_48a8_a979_e447011bc9c1.slice/crio-c4aa8613f315f4ab30cd6c67c86ee4ea13dbb4da299c903cc2aa6ffdaa97a07a WatchSource:0}: Error finding container c4aa8613f315f4ab30cd6c67c86ee4ea13dbb4da299c903cc2aa6ffdaa97a07a: Status 404 returned error can't find the container with id c4aa8613f315f4ab30cd6c67c86ee4ea13dbb4da299c903cc2aa6ffdaa97a07a
Nov 03 12:21:53.962882 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:53.962844    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7" event={"ID":"1d6f7294-7021-48a8-a979-e447011bc9c1","Type":"ContainerStarted","Data":"c4aa8613f315f4ab30cd6c67c86ee4ea13dbb4da299c903cc2aa6ffdaa97a07a"}
Nov 03 12:21:54.972047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:54.972001    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7" event={"ID":"1d6f7294-7021-48a8-a979-e447011bc9c1","Type":"ContainerStarted","Data":"1b320f25898bd52c6d0ff68165ec577ca99186abda64067fe20b1e46da790ec0"}
Nov 03 12:21:54.972486 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:54.972076    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:21:55.081324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:21:55.081277    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7" podStartSLOduration=2.3136001569999998 podStartE2EDuration="3.081263884s" totalImagesPullingTime="767.663727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:55.078074174 +0000 UTC m=+1864.090540837" watchObservedRunningTime="2026-11-03 12:21:55.081263884 +0000 UTC m=+1864.093730546"
Nov 03 12:22:00.984419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:00.984388    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:22:05.243471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:05.243435    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:22:09.564795 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:09.564764    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:22:21.516875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.516835    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"]
Nov 03 12:22:21.517353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.517056    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf" podUID="b7db405b-a248-4f5d-8f00-15f9cd647c57" containerName="maas-api" containerID="cri-o://d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89" gracePeriod=30
Nov 03 12:22:21.896969 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.896943    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:22:21.916311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.916274    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7db405b-a248-4f5d-8f00-15f9cd647c57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db405b-a248-4f5d-8f00-15f9cd647c57-openshift-service-ca\") pod \"b7db405b-a248-4f5d-8f00-15f9cd647c57\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") "
Nov 03 12:22:21.916459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.916380    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls\") pod \"b7db405b-a248-4f5d-8f00-15f9cd647c57\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") "
Nov 03 12:22:21.916501 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.916454    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7db405b-a248-4f5d-8f00-15f9cd647c57-kube-api-access-d56sb\" (UniqueName: \"kubernetes.io/projected/b7db405b-a248-4f5d-8f00-15f9cd647c57-kube-api-access-d56sb\") pod \"b7db405b-a248-4f5d-8f00-15f9cd647c57\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") "
Nov 03 12:22:21.916563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.916551    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-metrics-certs\") pod \"b7db405b-a248-4f5d-8f00-15f9cd647c57\" (UID: \"b7db405b-a248-4f5d-8f00-15f9cd647c57\") "
Nov 03 12:22:21.916745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.916676    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7db405b-a248-4f5d-8f00-15f9cd647c57-openshift-service-ca" pod "b7db405b-a248-4f5d-8f00-15f9cd647c57" (UID: "b7db405b-a248-4f5d-8f00-15f9cd647c57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:21.917151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.917130    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db405b-a248-4f5d-8f00-15f9cd647c57-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:21.918436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.918411    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls" pod "b7db405b-a248-4f5d-8f00-15f9cd647c57" (UID: "b7db405b-a248-4f5d-8f00-15f9cd647c57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:22:21.918624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.918607    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7db405b-a248-4f5d-8f00-15f9cd647c57-kube-api-access-d56sb" pod "b7db405b-a248-4f5d-8f00-15f9cd647c57" (UID: "b7db405b-a248-4f5d-8f00-15f9cd647c57"). InnerVolumeSpecName "kube-api-access-d56sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:21.918665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:21.918628    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-metrics-certs" pod "b7db405b-a248-4f5d-8f00-15f9cd647c57" (UID: "b7db405b-a248-4f5d-8f00-15f9cd647c57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:22:22.039138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.039105    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:22.039406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.039383    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d56sb\" (UniqueName: \"kubernetes.io/projected/b7db405b-a248-4f5d-8f00-15f9cd647c57-kube-api-access-d56sb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:22.039561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.039544    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db405b-a248-4f5d-8f00-15f9cd647c57-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:22.191447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.191412    4909 generic.go:356] "Generic (PLEG): container finished" podID="b7db405b-a248-4f5d-8f00-15f9cd647c57" containerID="d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89" exitCode=0
Nov 03 12:22:22.191630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.191462    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"
Nov 03 12:22:22.191630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.191494    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf" event={"ID":"b7db405b-a248-4f5d-8f00-15f9cd647c57","Type":"ContainerDied","Data":"d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89"}
Nov 03 12:22:22.191630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.191546    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf" event={"ID":"b7db405b-a248-4f5d-8f00-15f9cd647c57","Type":"ContainerDied","Data":"f94258375d809cbe076598e4cbdadb4046513b07a5c55fbf960ca141340701da"}
Nov 03 12:22:22.191630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.191569    4909 scope.go:122] "RemoveContainer" containerID="d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89"
Nov 03 12:22:22.202257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.202237    4909 scope.go:122] "RemoveContainer" containerID="d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89"
Nov 03 12:22:22.202514 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:22.202493    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89\": container with ID starting with d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89 not found: ID does not exist" containerID="d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89"
Nov 03 12:22:22.202582 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.202523    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89"} err="failed to get container status \"d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89\": rpc error: code = NotFound desc = could not find container \"d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89\": container with ID starting with d5d8f6104769aa5e2a165621692b5947d4e3ad7509579ac06fe8ca266a834a89 not found: ID does not exist"
Nov 03 12:22:22.344219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.344188    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"]
Nov 03 12:22:22.347070 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.347048    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-54zkf"]
Nov 03 12:22:22.997603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.997568    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"]
Nov 03 12:22:22.998238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:22.997772    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy" containerID="cri-o://c38b049e54cbd6cec887242123d90abaf4a03866bf37e5f55c9aff3383df2a82" gracePeriod=30
Nov 03 12:22:23.124307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:23.124275    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7db405b-a248-4f5d-8f00-15f9cd647c57" path="/var/lib/kubelet/pods/b7db405b-a248-4f5d-8f00-15f9cd647c57/volumes"
Nov 03 12:22:23.695992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:23.695959    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"]
Nov 03 12:22:23.697747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:23.697688    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy" containerID="cri-o://919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3" gracePeriod=30
Nov 03 12:22:23.723513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:23.723479    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"]
Nov 03 12:22:23.723780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:23.723752    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy" containerID="cri-o://5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2" gracePeriod=30
Nov 03 12:22:23.728525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:23.728493    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"]
Nov 03 12:22:23.728842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:23.728804    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy" containerID="cri-o://192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2" gracePeriod=30
Nov 03 12:22:28.237471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.237431    4909 generic.go:356] "Generic (PLEG): container finished" podID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerID="c38b049e54cbd6cec887242123d90abaf4a03866bf37e5f55c9aff3383df2a82" exitCode=0
Nov 03 12:22:28.237872 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.237483    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" event={"ID":"d91d447b-7696-4a75-9cdd-56dd77134e79","Type":"ContainerDied","Data":"c38b049e54cbd6cec887242123d90abaf4a03866bf37e5f55c9aff3383df2a82"}
Nov 03 12:22:28.293669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.293645    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:22:28.343667 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.343643    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-podinfo\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.343812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.343682    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-kube-api-access-vp8hs\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-kube-api-access-vp8hs\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.343812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.343742    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-certs\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.343812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.343796    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-data\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.344012 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.343992    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-credential-socket\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.344079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344058    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-token\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-token\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.344135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344085    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-data" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:28.344135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344091    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-envoy\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.344225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344104    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-certs" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:28.344280 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344180    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91d447b-7696-4a75-9cdd-56dd77134e79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91d447b-7696-4a75-9cdd-56dd77134e79-istiod-ca-cert\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.344334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344266    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-credential-socket" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:28.344334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344279    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-socket\") pod \"d91d447b-7696-4a75-9cdd-56dd77134e79\" (UID: \"d91d447b-7696-4a75-9cdd-56dd77134e79\") "
Nov 03 12:22:28.344463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344440    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91d447b-7696-4a75-9cdd-56dd77134e79-istiod-ca-cert" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:28.344589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344567    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-socket" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:28.344903 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344885    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.344985 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344909    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91d447b-7696-4a75-9cdd-56dd77134e79-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.344985 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344923    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.344985 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344936    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.344985 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.344949    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.345686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.345659    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-podinfo" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:28.346144 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.346124    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-token" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:28.346329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.346305    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-envoy" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:28.346329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.346306    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-kube-api-access-vp8hs" pod "d91d447b-7696-4a75-9cdd-56dd77134e79" (UID: "d91d447b-7696-4a75-9cdd-56dd77134e79"). InnerVolumeSpecName "kube-api-access-vp8hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:28.446356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.446327    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.446356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.446352    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8hs\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-kube-api-access-vp8hs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.446500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.446362    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.446500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.446371    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91d447b-7696-4a75-9cdd-56dd77134e79-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:28.988106 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:28.987258    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:22:29.021405 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.021369    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:22:29.054402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054376    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-credential-socket\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.054547 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054465    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-socket\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.054547 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054508    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-certs\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.054662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054550    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982d09ad-629f-49a9-9d7b-78a949dd600e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982d09ad-629f-49a9-9d7b-78a949dd600e-istiod-ca-cert\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.054662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054599    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-certs\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.054662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054637    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-kube-api-access-zprx6\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-kube-api-access-zprx6\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.054831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054668    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-credential-socket" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.054831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054676    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-podinfo\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.055039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054956    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-certs" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.055039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.054966    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-certs" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.055039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055000    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-socket" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.055039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055013    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-envoy\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.055302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055039    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982d09ad-629f-49a9-9d7b-78a949dd600e-istiod-ca-cert" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.055302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055183    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-envoy\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.055302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055247    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istiod-ca-cert\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.055302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055281    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-data\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.055552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055324    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-token\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-token\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.055552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055365    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-socket\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.055552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055414    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-token\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-token\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.055552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055465    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-podinfo\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.055552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055541    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-kube-api-access-hb7j9\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-kube-api-access-hb7j9\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.055830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055575    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-data\") pod \"982d09ad-629f-49a9-9d7b-78a949dd600e\" (UID: \"982d09ad-629f-49a9-9d7b-78a949dd600e\") "
Nov 03 12:22:29.055830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055627    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-credential-socket\") pod \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\" (UID: \"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d\") "
Nov 03 12:22:29.055830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055668    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istiod-ca-cert" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.055995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.055954    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-data" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.056189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056165    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-credential-socket" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.056189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056171    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-socket" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.056789 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056762    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982d09ad-629f-49a9-9d7b-78a949dd600e-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056796    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056813    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056827    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056841    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056854    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056866    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056879    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.056893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.056893    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.057342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.057122    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-data" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.057401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.057340    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-kube-api-access-zprx6" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "kube-api-access-zprx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.058940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.058918    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-podinfo" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.059077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.059059    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-kube-api-access-hb7j9" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "kube-api-access-hb7j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.059235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.059199    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-podinfo" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.059235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.059207    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-envoy" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.059403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.059386    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-token" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.059493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.059478    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-token" pod "982d09ad-629f-49a9-9d7b-78a949dd600e" (UID: "982d09ad-629f-49a9-9d7b-78a949dd600e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.060394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.060375    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-envoy" pod "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" (UID: "c9f26cd9-898e-4ab3-97af-e4adccdd4e1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.105483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.105463    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:22:29.157512 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157481    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-kube-api-access-9qr95\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-kube-api-access-9qr95\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.157629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157542    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-podinfo\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.157629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157601    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-data\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.157772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157694    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9044e50e-4ae3-4735-a37e-408156f3b9e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9044e50e-4ae3-4735-a37e-408156f3b9e1-istiod-ca-cert\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.157832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157821    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-token\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-token\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.157888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157859    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-certs\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.157947 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157907    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-envoy\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.157999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.157947    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-socket\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.158046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158021    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-credential-socket\") pod \"9044e50e-4ae3-4735-a37e-408156f3b9e1\" (UID: \"9044e50e-4ae3-4735-a37e-408156f3b9e1\") "
Nov 03 12:22:29.158169 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158127    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-data" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.158332 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158306    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9044e50e-4ae3-4735-a37e-408156f3b9e1-istiod-ca-cert" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.158412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158343    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-certs" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.158581 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158557    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-credential-socket" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.158581 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158570    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-socket" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.158723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158645    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158666    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158681    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158719    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158734    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158749    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb7j9\" (UniqueName: \"kubernetes.io/projected/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d-kube-api-access-hb7j9\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158765    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158779    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158792    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zprx6\" (UniqueName: \"kubernetes.io/projected/982d09ad-629f-49a9-9d7b-78a949dd600e-kube-api-access-zprx6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158805    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158819    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982d09ad-629f-49a9-9d7b-78a949dd600e-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.158938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.158831    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9044e50e-4ae3-4735-a37e-408156f3b9e1-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.160160 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.160128    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-kube-api-access-9qr95" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "kube-api-access-9qr95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.161040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.161007    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-envoy" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.161152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.161132    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-podinfo" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.161428 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.161402    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-token" pod "9044e50e-4ae3-4735-a37e-408156f3b9e1" (UID: "9044e50e-4ae3-4735-a37e-408156f3b9e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.246673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.246599    4909 generic.go:356] "Generic (PLEG): container finished" podID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerID="919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3" exitCode=0
Nov 03 12:22:29.247133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.246670    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"
Nov 03 12:22:29.247133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.246676    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" event={"ID":"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d","Type":"ContainerDied","Data":"919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3"}
Nov 03 12:22:29.247133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.246751    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2" event={"ID":"c9f26cd9-898e-4ab3-97af-e4adccdd4e1d","Type":"ContainerDied","Data":"13546aabaf9bbb38eaa692b3409cfbd2dbbb0dd5ad0ed6c7f7e44d5dc23ffe57"}
Nov 03 12:22:29.247133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.246785    4909 scope.go:122] "RemoveContainer" containerID="919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3"
Nov 03 12:22:29.249196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.249160    4909 generic.go:356] "Generic (PLEG): container finished" podID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerID="5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2" exitCode=0
Nov 03 12:22:29.249360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.249214    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" event={"ID":"982d09ad-629f-49a9-9d7b-78a949dd600e","Type":"ContainerDied","Data":"5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2"}
Nov 03 12:22:29.249427 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.249402    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg" event={"ID":"982d09ad-629f-49a9-9d7b-78a949dd600e","Type":"ContainerDied","Data":"9bfb3dc425a2983fc41d7cbd236ec6a0cdb884dbd22c3e789363c16aa01e961e"}
Nov 03 12:22:29.249487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.249225    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"
Nov 03 12:22:29.251887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.251869    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"
Nov 03 12:22:29.251992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.251901    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl" event={"ID":"d91d447b-7696-4a75-9cdd-56dd77134e79","Type":"ContainerDied","Data":"bae71a0ca31436115191344a4469410dd950b36ac9ab5511adbd41922e72cf36"}
Nov 03 12:22:29.254009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.253303    4909 generic.go:356] "Generic (PLEG): container finished" podID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerID="192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2" exitCode=0
Nov 03 12:22:29.254009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.253382    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" event={"ID":"9044e50e-4ae3-4735-a37e-408156f3b9e1","Type":"ContainerDied","Data":"192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2"}
Nov 03 12:22:29.254009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.253408    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd" event={"ID":"9044e50e-4ae3-4735-a37e-408156f3b9e1","Type":"ContainerDied","Data":"a1476c2cb34dbb9945337da5006fd21841d4d31be49d7a2d740c1f2a184a3e4c"}
Nov 03 12:22:29.254009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.253453    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"
Nov 03 12:22:29.261773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.261750    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.261868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.261777    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.261868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.261791    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.261868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.261804    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9044e50e-4ae3-4735-a37e-408156f3b9e1-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.261868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.261818    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qr95\" (UniqueName: \"kubernetes.io/projected/9044e50e-4ae3-4735-a37e-408156f3b9e1-kube-api-access-9qr95\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.261868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.261831    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9044e50e-4ae3-4735-a37e-408156f3b9e1-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.266898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.266882    4909 scope.go:122] "RemoveContainer" containerID="919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3"
Nov 03 12:22:29.267331 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:29.267305    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3\": container with ID starting with 919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3 not found: ID does not exist" containerID="919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3"
Nov 03 12:22:29.267874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.267342    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3"} err="failed to get container status \"919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3\": rpc error: code = NotFound desc = could not find container \"919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3\": container with ID starting with 919df9e544b5d8a69d62ae35478835461df80266bfb4da12aad4bac4043b76b3 not found: ID does not exist"
Nov 03 12:22:29.267874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.267365    4909 scope.go:122] "RemoveContainer" containerID="5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2"
Nov 03 12:22:29.281031 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.281014    4909 scope.go:122] "RemoveContainer" containerID="5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2"
Nov 03 12:22:29.281405 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:29.281381    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2\": container with ID starting with 5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2 not found: ID does not exist" containerID="5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2"
Nov 03 12:22:29.281488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.281412    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2"} err="failed to get container status \"5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2\": rpc error: code = NotFound desc = could not find container \"5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2\": container with ID starting with 5489ee14e17e2ec083a9b5295a65824de99efafd58663a8b5fac88c9a26275f2 not found: ID does not exist"
Nov 03 12:22:29.281488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.281432    4909 scope.go:122] "RemoveContainer" containerID="c38b049e54cbd6cec887242123d90abaf4a03866bf37e5f55c9aff3383df2a82"
Nov 03 12:22:29.291592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.291577    4909 scope.go:122] "RemoveContainer" containerID="192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2"
Nov 03 12:22:29.300857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.300839    4909 scope.go:122] "RemoveContainer" containerID="192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2"
Nov 03 12:22:29.301223 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:29.301195    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2\": container with ID starting with 192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2 not found: ID does not exist" containerID="192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2"
Nov 03 12:22:29.301341 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.301232    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2"} err="failed to get container status \"192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2\": rpc error: code = NotFound desc = could not find container \"192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2\": container with ID starting with 192b3dbf59597c6199238ac4b9cea2ec96722c21145802c2aa80dad30af209e2 not found: ID does not exist"
Nov 03 12:22:29.463249 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.463174    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"]
Nov 03 12:22:29.466304 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.466278    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-r8xtg"]
Nov 03 12:22:29.567624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.567536    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"]
Nov 03 12:22:29.572565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.572538    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m74wl"]
Nov 03 12:22:29.673145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.673116    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"]
Nov 03 12:22:29.678402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.678372    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-zvrd2"]
Nov 03 12:22:29.778009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.777980    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"]
Nov 03 12:22:29.781758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:29.781735    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7wtwd"]
Nov 03 12:22:31.127995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:31.127961    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" path="/var/lib/kubelet/pods/9044e50e-4ae3-4735-a37e-408156f3b9e1/volumes"
Nov 03 12:22:31.129335 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:31.129312    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" path="/var/lib/kubelet/pods/982d09ad-629f-49a9-9d7b-78a949dd600e/volumes"
Nov 03 12:22:31.130193 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:31.130175    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" path="/var/lib/kubelet/pods/c9f26cd9-898e-4ab3-97af-e4adccdd4e1d/volumes"
Nov 03 12:22:31.131418 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:31.131401    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" path="/var/lib/kubelet/pods/d91d447b-7696-4a75-9cdd-56dd77134e79/volumes"
Nov 03 12:22:36.376578 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:36.376542    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:22:39.250722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:39.249778    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:22:39.251238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:39.250944    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 03 12:22:39.282717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:39.281042    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:22:39.724519 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:39.724482    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:22:40.095935 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:40.095822    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" podUID="955970d0-cd2b-46b5-8d24-4cf06cf33989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:22:40.205722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:40.202773    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:22:40.309722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:40.308808    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:22:49.051420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.051385    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"]
Nov 03 12:22:49.051929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.051595    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv" podUID="ad58d7ea-9900-4ad0-a9b4-e303804b7031" containerName="maas-api" containerID="cri-o://6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea" gracePeriod=30
Nov 03 12:22:49.397281 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.397251    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:22:49.470631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.470601    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad58d7ea-9900-4ad0-a9b4-e303804b7031-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad58d7ea-9900-4ad0-a9b4-e303804b7031-openshift-service-ca\") pod \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") "
Nov 03 12:22:49.470825 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.470681    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad58d7ea-9900-4ad0-a9b4-e303804b7031-kube-api-access-gd8kh\" (UniqueName: \"kubernetes.io/projected/ad58d7ea-9900-4ad0-a9b4-e303804b7031-kube-api-access-gd8kh\") pod \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") "
Nov 03 12:22:49.470825 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.470737    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-maas-api-tls\") pod \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") "
Nov 03 12:22:49.470957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.470833    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-metrics-certs\") pod \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\" (UID: \"ad58d7ea-9900-4ad0-a9b4-e303804b7031\") "
Nov 03 12:22:49.471065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.471037    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad58d7ea-9900-4ad0-a9b4-e303804b7031-openshift-service-ca" pod "ad58d7ea-9900-4ad0-a9b4-e303804b7031" (UID: "ad58d7ea-9900-4ad0-a9b4-e303804b7031"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:49.471293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.471273    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad58d7ea-9900-4ad0-a9b4-e303804b7031-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:49.473056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.472900    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-metrics-certs" pod "ad58d7ea-9900-4ad0-a9b4-e303804b7031" (UID: "ad58d7ea-9900-4ad0-a9b4-e303804b7031"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:22:49.473056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.472972    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-maas-api-tls" pod "ad58d7ea-9900-4ad0-a9b4-e303804b7031" (UID: "ad58d7ea-9900-4ad0-a9b4-e303804b7031"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:22:49.473056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.472999    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad58d7ea-9900-4ad0-a9b4-e303804b7031-kube-api-access-gd8kh" pod "ad58d7ea-9900-4ad0-a9b4-e303804b7031" (UID: "ad58d7ea-9900-4ad0-a9b4-e303804b7031"). InnerVolumeSpecName "kube-api-access-gd8kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:49.528814 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.528784    4909 generic.go:356] "Generic (PLEG): container finished" podID="ad58d7ea-9900-4ad0-a9b4-e303804b7031" containerID="6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea" exitCode=0
Nov 03 12:22:49.528957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.528842    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"
Nov 03 12:22:49.528957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.528875    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv" event={"ID":"ad58d7ea-9900-4ad0-a9b4-e303804b7031","Type":"ContainerDied","Data":"6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea"}
Nov 03 12:22:49.529049 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.528971    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv" event={"ID":"ad58d7ea-9900-4ad0-a9b4-e303804b7031","Type":"ContainerDied","Data":"3b2903351b8af10a24c8a53e3d479a1cca62cd5d51ae716f59e63bb1ee2ae6f7"}
Nov 03 12:22:49.529049 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.529003    4909 scope.go:122] "RemoveContainer" containerID="6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea"
Nov 03 12:22:49.541663 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.541632    4909 scope.go:122] "RemoveContainer" containerID="6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea"
Nov 03 12:22:49.542008 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:49.541981    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea\": container with ID starting with 6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea not found: ID does not exist" containerID="6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea"
Nov 03 12:22:49.542112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.542019    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea"} err="failed to get container status \"6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea\": rpc error: code = NotFound desc = could not find container \"6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea\": container with ID starting with 6f02830ec988337b4f2628d83e7966aa70ee136cf18a683904b560d7f545a3ea not found: ID does not exist"
Nov 03 12:22:49.572767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.572735    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd8kh\" (UniqueName: \"kubernetes.io/projected/ad58d7ea-9900-4ad0-a9b4-e303804b7031-kube-api-access-gd8kh\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:49.572767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.572773    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:49.572979 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.572790    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad58d7ea-9900-4ad0-a9b4-e303804b7031-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:49.683555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.683454    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"]
Nov 03 12:22:49.686819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:49.686794    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv"]
Nov 03 12:22:50.569520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.569486    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"]
Nov 03 12:22:50.570034 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570021    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b7db405b-a248-4f5d-8f00-15f9cd647c57" containerName="maas-api"
Nov 03 12:22:50.570078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570036    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7db405b-a248-4f5d-8f00-15f9cd647c57" containerName="maas-api"
Nov 03 12:22:50.570078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570046    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy"
Nov 03 12:22:50.570078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570051    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy"
Nov 03 12:22:50.570078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570072    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy"
Nov 03 12:22:50.570078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570078    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy"
Nov 03 12:22:50.570221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570090    4909 cpu_manager.go:414] "containerMap: removing container" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy"
Nov 03 12:22:50.570221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570095    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy"
Nov 03 12:22:50.570221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570114    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ad58d7ea-9900-4ad0-a9b4-e303804b7031" containerName="maas-api"
Nov 03 12:22:50.570221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570119    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad58d7ea-9900-4ad0-a9b4-e303804b7031" containerName="maas-api"
Nov 03 12:22:50.570221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570136    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy"
Nov 03 12:22:50.570221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570140    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy"
Nov 03 12:22:50.570221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570221    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad58d7ea-9900-4ad0-a9b4-e303804b7031" containerName="maas-api"
Nov 03 12:22:50.570423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570232    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9044e50e-4ae3-4735-a37e-408156f3b9e1" containerName="istio-proxy"
Nov 03 12:22:50.570423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570238    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7db405b-a248-4f5d-8f00-15f9cd647c57" containerName="maas-api"
Nov 03 12:22:50.570423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570244    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f26cd9-898e-4ab3-97af-e4adccdd4e1d" containerName="istio-proxy"
Nov 03 12:22:50.570423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570252    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91d447b-7696-4a75-9cdd-56dd77134e79" containerName="istio-proxy"
Nov 03 12:22:50.570423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.570258    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="982d09ad-629f-49a9-9d7b-78a949dd600e" containerName="istio-proxy"
Nov 03 12:22:50.575369 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.575351    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.627295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.627259    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"]
Nov 03 12:22:50.664326 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.664293    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.664528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.664507    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rm7\" (UniqueName: \"kubernetes.io/projected/72662b2c-2f65-479e-8e81-a41ab7e24468-kube-api-access-j8rm7\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.664606 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.664548    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72662b2c-2f65-479e-8e81-a41ab7e24468-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.664668 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.664629    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-metrics-certs\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.669224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.669206    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-059838-serving-cert\""
Nov 03 12:22:50.765393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.765352    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rm7\" (UniqueName: \"kubernetes.io/projected/72662b2c-2f65-479e-8e81-a41ab7e24468-kube-api-access-j8rm7\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.765572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.765412    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72662b2c-2f65-479e-8e81-a41ab7e24468-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.765572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.765472    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-metrics-certs\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.765688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.765596    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.765789 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:50.765773    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-serving-cert: secret "maas-api-e2e-ipp-b-059838-serving-cert" not found
Nov 03 12:22:50.765862 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:22:50.765850    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls podName:72662b2c-2f65-479e-8e81-a41ab7e24468 nodeName:}" failed. No retries permitted until 2026-11-03 12:22:51.265831697 +0000 UTC m=+1920.278298349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls") pod "maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" (UID: "72662b2c-2f65-479e-8e81-a41ab7e24468") : secret "maas-api-e2e-ipp-b-059838-serving-cert" not found
Nov 03 12:22:50.766188 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.766162    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72662b2c-2f65-479e-8e81-a41ab7e24468-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.768030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.768012    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-metrics-certs\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:50.818377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:50.818343    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rm7\" (UniqueName: \"kubernetes.io/projected/72662b2c-2f65-479e-8e81-a41ab7e24468-kube-api-access-j8rm7\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:51.124126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:51.124098    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad58d7ea-9900-4ad0-a9b4-e303804b7031" path="/var/lib/kubelet/pods/ad58d7ea-9900-4ad0-a9b4-e303804b7031/volumes"
Nov 03 12:22:51.280653 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:51.280623    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:51.283149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:51.283121    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls\") pod \"maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:51.488694 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:51.488665    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:51.859981 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:51.859953    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"]
Nov 03 12:22:51.861631 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:22:51.861600    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72662b2c_2f65_479e_8e81_a41ab7e24468.slice/crio-ded420e025d4a20cb0687ce9cfc4034a70413c9c58cf4811f28887861b0185de WatchSource:0}: Error finding container ded420e025d4a20cb0687ce9cfc4034a70413c9c58cf4811f28887861b0185de: Status 404 returned error can't find the container with id ded420e025d4a20cb0687ce9cfc4034a70413c9c58cf4811f28887861b0185de
Nov 03 12:22:52.296143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:52.296107    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e8d12-cd5cbc546-k7kkv" podUID="ad58d7ea-9900-4ad0-a9b4-e303804b7031" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.96:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:22:52.594065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:52.593967    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" event={"ID":"72662b2c-2f65-479e-8e81-a41ab7e24468","Type":"ContainerStarted","Data":"ded420e025d4a20cb0687ce9cfc4034a70413c9c58cf4811f28887861b0185de"}
Nov 03 12:22:53.603385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:53.603352    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" event={"ID":"72662b2c-2f65-479e-8e81-a41ab7e24468","Type":"ContainerStarted","Data":"817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95"}
Nov 03 12:22:53.603828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:53.603440    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:22:53.715593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:53.715540    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" podStartSLOduration=2.832144047 podStartE2EDuration="3.715526921s" totalImagesPullingTime="883.382874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:22:53.712230695 +0000 UTC m=+1922.724697358" watchObservedRunningTime="2026-11-03 12:22:53.715526921 +0000 UTC m=+1922.727993604"
Nov 03 12:22:59.625924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:22:59.625891    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:23:09.563455 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:23:09.563378    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:23:11.250330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:11.250292    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"]
Nov 03 12:23:11.250936 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:11.250560    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy" containerID="cri-o://b05ed1cbec0304bc1b912f17a9f9dca4469d27012788a6b175fb956cebe4aff1" gracePeriod=30
Nov 03 12:23:11.557315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:11.557123    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"]
Nov 03 12:23:11.558784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:11.558656    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy" containerID="cri-o://33a5d6e8a0bca22ae18299d8afb27963d19070a8d8752280c3c7a721f38b10d3" gracePeriod=30
Nov 03 12:23:11.928469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:11.928383    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:23:16.197838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:16.197794    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:23:16.198452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:16.198418    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.714725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:16.692229    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:17.144271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.144181    4909 generic.go:356] "Generic (PLEG): container finished" podID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerID="33a5d6e8a0bca22ae18299d8afb27963d19070a8d8752280c3c7a721f38b10d3" exitCode=0
Nov 03 12:23:17.144632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.144606    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" event={"ID":"12d9fb72-0575-4a4c-a47d-fd875216453c","Type":"ContainerDied","Data":"33a5d6e8a0bca22ae18299d8afb27963d19070a8d8752280c3c7a721f38b10d3"}
Nov 03 12:23:17.144817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.144801    4909 scope.go:122] "RemoveContainer" containerID="c3b45b148eedde5319edd61885d7e8541b20818353e73bbadbc5242c111d136b"
Nov 03 12:23:17.149614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.149592    4909 generic.go:356] "Generic (PLEG): container finished" podID="3a0642be-2076-4628-8206-686e571db79c" containerID="b05ed1cbec0304bc1b912f17a9f9dca4469d27012788a6b175fb956cebe4aff1" exitCode=0
Nov 03 12:23:17.149810 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.149790    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" event={"ID":"3a0642be-2076-4628-8206-686e571db79c","Type":"ContainerDied","Data":"b05ed1cbec0304bc1b912f17a9f9dca4469d27012788a6b175fb956cebe4aff1"}
Nov 03 12:23:17.290010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.289975    4909 scope.go:122] "RemoveContainer" containerID="68bb5867fdb4a4bc87974129f554e000b7e575639e6e7aafbbd2c010b9925be1"
Nov 03 12:23:17.338759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.338681    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:23:17.412182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412154    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0642be-2076-4628-8206-686e571db79c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0642be-2076-4628-8206-686e571db79c-istiod-ca-cert\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412207    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-socket\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412277    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-istio-token\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-istio-token\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412367    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a0642be-2076-4628-8206-686e571db79c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0642be-2076-4628-8206-686e571db79c-istio-podinfo\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412549    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-socket" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.412577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412566    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0642be-2076-4628-8206-686e571db79c-istiod-ca-cert" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:17.412731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412612    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-data\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412687    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-credential-socket\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412792    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-certs\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412898    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-kube-api-access-jm97f\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-kube-api-access-jm97f\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.412968 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412946    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-envoy\") pod \"3a0642be-2076-4628-8206-686e571db79c\" (UID: \"3a0642be-2076-4628-8206-686e571db79c\") "
Nov 03 12:23:17.413021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.412978    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-data" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.413064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.413016    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-credential-socket" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.413064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.413057    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-certs" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.413489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.413465    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.413601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.413491    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.413601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.413508    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.413601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.413522    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0642be-2076-4628-8206-686e571db79c-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.413601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.413535    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.415120 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.415097    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-envoy" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.415218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.415165    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-kube-api-access-jm97f" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "kube-api-access-jm97f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:17.415272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.415230    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a0642be-2076-4628-8206-686e571db79c-istio-podinfo" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:23:17.415272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.415239    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:23:17.415451 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.415433    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-istio-token" pod "3a0642be-2076-4628-8206-686e571db79c" (UID: "3a0642be-2076-4628-8206-686e571db79c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:17.513966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.513924    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-token\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-token\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514012    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-kube-api-access-sbmmm\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-kube-api-access-sbmmm\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514049    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-data\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514108    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-certs\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514148    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-credential-socket\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514248    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-envoy\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514358    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-socket\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514412    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-data" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.514504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514444    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d9fb72-0575-4a4c-a47d-fd875216453c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d9fb72-0575-4a4c-a47d-fd875216453c-istiod-ca-cert\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514472    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-podinfo\") pod \"12d9fb72-0575-4a4c-a47d-fd875216453c\" (UID: \"12d9fb72-0575-4a4c-a47d-fd875216453c\") "
Nov 03 12:23:17.514662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514630    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-socket" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.514909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514797    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-certs" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.514909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.514882    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d9fb72-0575-4a4c-a47d-fd875216453c-istiod-ca-cert" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:17.515130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515104    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515133    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0642be-2076-4628-8206-686e571db79c-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515148    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515162    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d9fb72-0575-4a4c-a47d-fd875216453c-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515176    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm97f\" (UniqueName: \"kubernetes.io/projected/3a0642be-2076-4628-8206-686e571db79c-kube-api-access-jm97f\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515191    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0642be-2076-4628-8206-686e571db79c-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515205    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515218    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.515527 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.515287    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-credential-socket" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.516905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.516872    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-token" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:17.517263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.517238    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-kube-api-access-sbmmm" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "kube-api-access-sbmmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:17.517491 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.517467    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-envoy" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.518127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.518103    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-podinfo" pod "12d9fb72-0575-4a4c-a47d-fd875216453c" (UID: "12d9fb72-0575-4a4c-a47d-fd875216453c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:23:17.617313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.617230    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.617313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.617267    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.617313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.617283    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmmm\" (UniqueName: \"kubernetes.io/projected/12d9fb72-0575-4a4c-a47d-fd875216453c-kube-api-access-sbmmm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.617313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.617298    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.617313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:17.617311    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d9fb72-0575-4a4c-a47d-fd875216453c-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:18.158308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.158272    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"
Nov 03 12:23:18.158502 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.158333    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh" event={"ID":"3a0642be-2076-4628-8206-686e571db79c","Type":"ContainerDied","Data":"8321ea9b8bc78ad8a4c3178272a31bb992911a821ff2be656f92848f07d4386a"}
Nov 03 12:23:18.158502 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.158386    4909 scope.go:122] "RemoveContainer" containerID="b05ed1cbec0304bc1b912f17a9f9dca4469d27012788a6b175fb956cebe4aff1"
Nov 03 12:23:18.159873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.159848    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j" event={"ID":"12d9fb72-0575-4a4c-a47d-fd875216453c","Type":"ContainerDied","Data":"39abe6b39a4509f414ae748e86a857b63b362bea2117c6dcddb03ef6db42f52e"}
Nov 03 12:23:18.159873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.159867    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"
Nov 03 12:23:18.168917 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.168901    4909 scope.go:122] "RemoveContainer" containerID="33a5d6e8a0bca22ae18299d8afb27963d19070a8d8752280c3c7a721f38b10d3"
Nov 03 12:23:18.356795 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.356761    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"]
Nov 03 12:23:18.403664 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.403634    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45b1b7-openshift-default-dfcc95bf9-2d9bh"]
Nov 03 12:23:18.540033 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.532289    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"]
Nov 03 12:23:18.540591 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:18.540488    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-h2f7j"]
Nov 03 12:23:19.123202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:19.123169    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" path="/var/lib/kubelet/pods/12d9fb72-0575-4a4c-a47d-fd875216453c/volumes"
Nov 03 12:23:19.123745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:19.123728    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0642be-2076-4628-8206-686e571db79c" path="/var/lib/kubelet/pods/3a0642be-2076-4628-8206-686e571db79c/volumes"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.750872    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"]
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751693    4909 cpu_manager.go:414] "containerMap: removing container" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751743    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751762    4909 cpu_manager.go:414] "containerMap: removing container" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751771    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751783    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751792    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751975    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy"
Nov 03 12:23:22.754740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.751991    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy"
Nov 03 12:23:22.788793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.788302    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:22.819088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.819060    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"]
Nov 03 12:23:22.830564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.830527    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f977abf-28a1-4ba9-8d00-9845c7671b92-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:22.830760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.830743    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lkv\" (UniqueName: \"kubernetes.io/projected/2f977abf-28a1-4ba9-8d00-9845c7671b92-kube-api-access-l9lkv\") pod \"payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:22.860932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.860912    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c8ee66\""
Nov 03 12:23:22.861039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.861002    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c8ee66-dockercfg-272br\""
Nov 03 12:23:22.931416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.931393    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f977abf-28a1-4ba9-8d00-9845c7671b92-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:22.931533 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.931518    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lkv\" (UniqueName: \"kubernetes.io/projected/2f977abf-28a1-4ba9-8d00-9845c7671b92-kube-api-access-l9lkv\") pod \"payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:22.932122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.932102    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f977abf-28a1-4ba9-8d00-9845c7671b92-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:22.984270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:22.984246    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lkv\" (UniqueName: \"kubernetes.io/projected/2f977abf-28a1-4ba9-8d00-9845c7671b92-kube-api-access-l9lkv\") pod \"payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:23.115866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:23.115806    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:23.484492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:23.484469    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"]
Nov 03 12:23:23.485935 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:23:23.485896    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f977abf_28a1_4ba9_8d00_9845c7671b92.slice/crio-a5203247c16062a59c0b70e158ac6e00fb4954587baeab7d408f9af2a633d511 WatchSource:0}: Error finding container a5203247c16062a59c0b70e158ac6e00fb4954587baeab7d408f9af2a633d511: Status 404 returned error can't find the container with id a5203247c16062a59c0b70e158ac6e00fb4954587baeab7d408f9af2a633d511
Nov 03 12:23:24.213564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:24.213530    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg" event={"ID":"2f977abf-28a1-4ba9-8d00-9845c7671b92","Type":"ContainerStarted","Data":"5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f"}
Nov 03 12:23:24.213980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:24.213594    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:24.213980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:24.213611    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg" event={"ID":"2f977abf-28a1-4ba9-8d00-9845c7671b92","Type":"ContainerStarted","Data":"a5203247c16062a59c0b70e158ac6e00fb4954587baeab7d408f9af2a633d511"}
Nov 03 12:23:24.320770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:24.320721    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg" podStartSLOduration=2.320692229 podStartE2EDuration="2.320692229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:24.318167802 +0000 UTC m=+1953.330634465" watchObservedRunningTime="2026-11-03 12:23:24.320692229 +0000 UTC m=+1953.333158893"
Nov 03 12:23:35.223588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:35.223560    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:23:37.603367 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.603333    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"]
Nov 03 12:23:37.603933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.603918    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy"
Nov 03 12:23:37.603933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.603934    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy"
Nov 03 12:23:37.604073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.604059    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0642be-2076-4628-8206-686e571db79c" containerName="istio-proxy"
Nov 03 12:23:37.604133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.604084    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d9fb72-0575-4a4c-a47d-fd875216453c" containerName="istio-proxy"
Nov 03 12:23:37.608301 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.608282    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.656467 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.656433    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d25504-openshift-default-dockercfg-7qtfh\""
Nov 03 12:23:37.706737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.706679    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"]
Nov 03 12:23:37.722325 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722295    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-certs\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722477 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722337    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-credential-socket\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722477 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722412    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-token\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722500    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istiod-ca-cert\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722582    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-socket\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722712 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722628    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-data\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722712 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722664    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-podinfo\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722810 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722780    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4bj\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-kube-api-access-bn4bj\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.722900 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.722885    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-envoy\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.859765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.859529    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-envoy\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.859765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.859683    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-certs\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.859765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.859751    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-credential-socket\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.860041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.859808    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-token\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.860041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.859918    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istiod-ca-cert\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.860041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.859968    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-socket\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.860041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.860010    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-data\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.860257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.860047    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-podinfo\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.860257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.860081    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4bj\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-kube-api-access-bn4bj\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.867718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.862226    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istiod-ca-cert\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.867718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.862471    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-certs\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.875716 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.872125    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-data\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.875716 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.872374    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-socket\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.875716 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.872807    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-credential-socket\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.881722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.877025    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-envoy\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.891757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.884629    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-podinfo\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.934730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.925018    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4bj\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-kube-api-access-bn4bj\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:37.937410 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:37.937366    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-token\") pod \"e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") " pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:38.221253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:38.221226    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:38.638918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:38.638889    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"]
Nov 03 12:23:38.640134 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:23:38.640107    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e6b8e3_35ed_4714_b43e_a63d7267daa2.slice/crio-cbc4f7fffa1b7e22c25beaf5038482db0a6a3b08d9d12cfce282a34668966676 WatchSource:0}: Error finding container cbc4f7fffa1b7e22c25beaf5038482db0a6a3b08d9d12cfce282a34668966676: Status 404 returned error can't find the container with id cbc4f7fffa1b7e22c25beaf5038482db0a6a3b08d9d12cfce282a34668966676
Nov 03 12:23:38.642476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:38.642443    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:23:38.642567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:38.642510    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:23:38.642567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:38.642538    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:23:39.311383 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:39.311352    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" event={"ID":"f3e6b8e3-35ed-4714-b43e-a63d7267daa2","Type":"ContainerStarted","Data":"d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f"}
Nov 03 12:23:39.311553 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:39.311403    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" event={"ID":"f3e6b8e3-35ed-4714-b43e-a63d7267daa2","Type":"ContainerStarted","Data":"cbc4f7fffa1b7e22c25beaf5038482db0a6a3b08d9d12cfce282a34668966676"}
Nov 03 12:23:39.419018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:39.418970    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podStartSLOduration=2.41895735 podStartE2EDuration="2.41895735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:39.417837391 +0000 UTC m=+1968.430304068" watchObservedRunningTime="2026-11-03 12:23:39.41895735 +0000 UTC m=+1968.431424013"
Nov 03 12:23:39.606964 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:23:39.606864    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:23:40.222115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:40.222083    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:40.228259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:40.228227    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:40.318846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:40.318810    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:40.319811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:40.319786    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:23:42.593799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:42.593756    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"]
Nov 03 12:23:42.594373 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:42.594006    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" podUID="5c18ee25-cbd5-446a-a2c7-c3487a838cc0" containerName="maas-api" containerID="cri-o://083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc" gracePeriod=30
Nov 03 12:23:42.930521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:42.930497    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:23:43.033410 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.033378    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-kube-api-access-522c9\" (UniqueName: \"kubernetes.io/projected/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-kube-api-access-522c9\") pod \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") "
Nov 03 12:23:43.033564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.033419    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-metrics-certs\") pod \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") "
Nov 03 12:23:43.033629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.033568    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-openshift-service-ca\") pod \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") "
Nov 03 12:23:43.033742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.033687    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls\") pod \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\" (UID: \"5c18ee25-cbd5-446a-a2c7-c3487a838cc0\") "
Nov 03 12:23:43.033990 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.033946    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-openshift-service-ca" pod "5c18ee25-cbd5-446a-a2c7-c3487a838cc0" (UID: "5c18ee25-cbd5-446a-a2c7-c3487a838cc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:43.034298 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.034276    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.035881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.035858    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-metrics-certs" pod "5c18ee25-cbd5-446a-a2c7-c3487a838cc0" (UID: "5c18ee25-cbd5-446a-a2c7-c3487a838cc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:23:43.036112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.036094    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-kube-api-access-522c9" pod "5c18ee25-cbd5-446a-a2c7-c3487a838cc0" (UID: "5c18ee25-cbd5-446a-a2c7-c3487a838cc0"). InnerVolumeSpecName "kube-api-access-522c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:43.036325 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.036294    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls" pod "5c18ee25-cbd5-446a-a2c7-c3487a838cc0" (UID: "5c18ee25-cbd5-446a-a2c7-c3487a838cc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:23:43.136734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.136080    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.136734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.136110    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-522c9\" (UniqueName: \"kubernetes.io/projected/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-kube-api-access-522c9\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.136734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.136123    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c18ee25-cbd5-446a-a2c7-c3487a838cc0-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.338069 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.338035    4909 generic.go:356] "Generic (PLEG): container finished" podID="5c18ee25-cbd5-446a-a2c7-c3487a838cc0" containerID="083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc" exitCode=0
Nov 03 12:23:43.338237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.338085    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"
Nov 03 12:23:43.338237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.338122    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" event={"ID":"5c18ee25-cbd5-446a-a2c7-c3487a838cc0","Type":"ContainerDied","Data":"083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc"}
Nov 03 12:23:43.338237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.338188    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" event={"ID":"5c18ee25-cbd5-446a-a2c7-c3487a838cc0","Type":"ContainerDied","Data":"b8b603fce0ce42e2c66fa0fe6a8b8ef786c63ff9c978109747dcf218666bf0dd"}
Nov 03 12:23:43.338237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.338220    4909 scope.go:122] "RemoveContainer" containerID="083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc"
Nov 03 12:23:43.347970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.347955    4909 scope.go:122] "RemoveContainer" containerID="083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc"
Nov 03 12:23:43.348250 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:23:43.348226    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc\": container with ID starting with 083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc not found: ID does not exist" containerID="083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc"
Nov 03 12:23:43.348350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.348254    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc"} err="failed to get container status \"083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc\": rpc error: code = NotFound desc = could not find container \"083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc\": container with ID starting with 083298a6c9db8d4772c7b8b4e35835f1b1e8f5987b2fbbf16c541e573095b1dc not found: ID does not exist"
Nov 03 12:23:43.491227 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.491193    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"]
Nov 03 12:23:43.493955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:43.493922    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4"]
Nov 03 12:23:45.124995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:45.124962    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c18ee25-cbd5-446a-a2c7-c3487a838cc0" path="/var/lib/kubelet/pods/5c18ee25-cbd5-446a-a2c7-c3487a838cc0/volumes"
Nov 03 12:23:45.840448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:45.838953    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1e151c-7fb54c4c46-q6pf4" podUID="5c18ee25-cbd5-446a-a2c7-c3487a838cc0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.102:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:47.412379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:23:47.412347    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:24:08.606171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:08.606141    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"]
Nov 03 12:24:08.606613 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:08.606361    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" containerID="cri-o://28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402" gracePeriod=30
Nov 03 12:24:09.565304 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:09.565267    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:24:11.258406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.258372    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"]
Nov 03 12:24:11.259253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.259225    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5c18ee25-cbd5-446a-a2c7-c3487a838cc0" containerName="maas-api"
Nov 03 12:24:11.259345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.259258    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c18ee25-cbd5-446a-a2c7-c3487a838cc0" containerName="maas-api"
Nov 03 12:24:11.259507 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.259473    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c18ee25-cbd5-446a-a2c7-c3487a838cc0" containerName="maas-api"
Nov 03 12:24:11.264824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.264805    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.296487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.296462    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.296585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.296496    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c658c628-1456-4e92-a3aa-12a12832e307-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.296630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.296617    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-metrics-certs\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.296661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.296644    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mpr\" (UniqueName: \"kubernetes.io/projected/c658c628-1456-4e92-a3aa-12a12832e307-kube-api-access-86mpr\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.314828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.314802    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"]
Nov 03 12:24:11.354014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.353984    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c8ee66-serving-cert\""
Nov 03 12:24:11.398010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.397984    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.398163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.398018    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c658c628-1456-4e92-a3aa-12a12832e307-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.398163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.398119    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-metrics-certs\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.398286 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:11.398161    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-serving-cert: secret "maas-api-e2e-praxis-a-c8ee66-serving-cert" not found
Nov 03 12:24:11.398286 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:11.398239    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls podName:c658c628-1456-4e92-a3aa-12a12832e307 nodeName:}" failed. No retries permitted until 2026-11-03 12:24:11.898219098 +0000 UTC m=+2000.910685755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls") pod "maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm" (UID: "c658c628-1456-4e92-a3aa-12a12832e307") : secret "maas-api-e2e-praxis-a-c8ee66-serving-cert" not found
Nov 03 12:24:11.398449 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.398163    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mpr\" (UniqueName: \"kubernetes.io/projected/c658c628-1456-4e92-a3aa-12a12832e307-kube-api-access-86mpr\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.398999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.398979    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c658c628-1456-4e92-a3aa-12a12832e307-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.401221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.401198    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-metrics-certs\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.453361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.453336    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mpr\" (UniqueName: \"kubernetes.io/projected/c658c628-1456-4e92-a3aa-12a12832e307-kube-api-access-86mpr\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.664370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.664284    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:24:11.664646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.664614    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:24:11.712404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.705522    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"]
Nov 03 12:24:11.712404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.705774    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy" containerID="cri-o://1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125" gracePeriod=30
Nov 03 12:24:11.917084 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.917005    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:11.919577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:11.919544    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:12.179469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:12.179385    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:12.547791 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:12.547750    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"]
Nov 03 12:24:12.549449 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:24:12.549418    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc658c628_1456_4e92_a3aa_12a12832e307.slice/crio-f1ef08a6dab2c0c05fd1dc5ca1db87b82373a7c407850af17fee6653273416dd WatchSource:0}: Error finding container f1ef08a6dab2c0c05fd1dc5ca1db87b82373a7c407850af17fee6653273416dd: Status 404 returned error can't find the container with id f1ef08a6dab2c0c05fd1dc5ca1db87b82373a7c407850af17fee6653273416dd
Nov 03 12:24:12.579904 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:12.579872    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm" event={"ID":"c658c628-1456-4e92-a3aa-12a12832e307","Type":"ContainerStarted","Data":"f1ef08a6dab2c0c05fd1dc5ca1db87b82373a7c407850af17fee6653273416dd"}
Nov 03 12:24:13.587601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:13.587570    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm" event={"ID":"c658c628-1456-4e92-a3aa-12a12832e307","Type":"ContainerStarted","Data":"1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9"}
Nov 03 12:24:13.588042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:13.587687    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:13.720122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:13.720034    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm" podStartSLOduration=1.809891731 podStartE2EDuration="2.720020073s" totalImagesPullingTime="910.128342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:24:13.719519619 +0000 UTC m=+2002.731986312" watchObservedRunningTime="2026-11-03 12:24:13.720020073 +0000 UTC m=+2002.732486736"
Nov 03 12:24:13.921713 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:13.921669    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:24:14.055222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.055136    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-socket\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.055496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.055477    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-istio-token\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-istio-token\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.055638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.055622    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-data\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.055789 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.055774    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b851622c-afe9-4f41-9c18-0e04404289ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b851622c-afe9-4f41-9c18-0e04404289ed-istio-podinfo\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.055918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.055903    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-envoy\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.056068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.056055    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-kube-api-access-rmxwj\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-kube-api-access-rmxwj\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.056228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.056215    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-certs\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.056399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.056386    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b851622c-afe9-4f41-9c18-0e04404289ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b851622c-afe9-4f41-9c18-0e04404289ed-istiod-ca-cert\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.056538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.056525    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-credential-socket\") pod \"b851622c-afe9-4f41-9c18-0e04404289ed\" (UID: \"b851622c-afe9-4f41-9c18-0e04404289ed\") "
Nov 03 12:24:14.057410 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.057385    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-credential-socket" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:14.057786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.057765    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-socket" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:14.062894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.062867    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-data" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:14.063212 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.063193    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-certs" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:14.064247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.064222    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b851622c-afe9-4f41-9c18-0e04404289ed-istiod-ca-cert" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:14.082367 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.082330    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-istio-token" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:14.082921 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.082891    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-envoy" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:14.089288 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.089258    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b851622c-afe9-4f41-9c18-0e04404289ed-istio-podinfo" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:24:14.104061 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.102028    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-kube-api-access-rmxwj" pod "b851622c-afe9-4f41-9c18-0e04404289ed" (UID: "b851622c-afe9-4f41-9c18-0e04404289ed"). InnerVolumeSpecName "kube-api-access-rmxwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:14.158565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158527    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158560    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b851622c-afe9-4f41-9c18-0e04404289ed-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158569    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158578    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158587    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158595    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158603    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b851622c-afe9-4f41-9c18-0e04404289ed-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158612    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b851622c-afe9-4f41-9c18-0e04404289ed-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.158894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.158621    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmxwj\" (UniqueName: \"kubernetes.io/projected/b851622c-afe9-4f41-9c18-0e04404289ed-kube-api-access-rmxwj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:14.595662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.595627    4909 generic.go:356] "Generic (PLEG): container finished" podID="b851622c-afe9-4f41-9c18-0e04404289ed" containerID="28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402" exitCode=0
Nov 03 12:24:14.596119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.595688    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"
Nov 03 12:24:14.596119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.595725    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" event={"ID":"b851622c-afe9-4f41-9c18-0e04404289ed","Type":"ContainerDied","Data":"28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402"}
Nov 03 12:24:14.596119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.595780    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs" event={"ID":"b851622c-afe9-4f41-9c18-0e04404289ed","Type":"ContainerDied","Data":"e8169f70069f61015afff0cbd5a32e90f77480b818fa3053a198a55f7f61129f"}
Nov 03 12:24:14.596119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.595804    4909 scope.go:122] "RemoveContainer" containerID="28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402"
Nov 03 12:24:14.605398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.605382    4909 scope.go:122] "RemoveContainer" containerID="28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402"
Nov 03 12:24:14.605653 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:14.605633    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402\": container with ID starting with 28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402 not found: ID does not exist" containerID="28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402"
Nov 03 12:24:14.605762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.605660    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402"} err="failed to get container status \"28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402\": rpc error: code = NotFound desc = could not find container \"28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402\": container with ID starting with 28cf42c9cc6f02047ff3735175d9f491f7b21861a579656d0cf960aab36c1402 not found: ID does not exist"
Nov 03 12:24:14.751489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.751456    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"]
Nov 03 12:24:14.755672 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:14.755650    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gpjfs"]
Nov 03 12:24:15.123767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:15.123736    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" path="/var/lib/kubelet/pods/b851622c-afe9-4f41-9c18-0e04404289ed/volumes"
Nov 03 12:24:17.388152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.388130    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:24:17.496370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496297    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-credential-socket\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.496370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496358    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-podinfo\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.496592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496389    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf01078b-da80-420e-b7ff-8a6c1febca7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf01078b-da80-420e-b7ff-8a6c1febca7e-istiod-ca-cert\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.496592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496418    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-socket\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.496592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496448    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-kube-api-access-lng2v\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-kube-api-access-lng2v\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.496592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496520    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-envoy\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.496592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496577    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-credential-socket" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.497011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496584    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-certs\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.497011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496683    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-token\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-token\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.497011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496760    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-socket" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.497011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496777    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-data\") pod \"bf01078b-da80-420e-b7ff-8a6c1febca7e\" (UID: \"bf01078b-da80-420e-b7ff-8a6c1febca7e\") "
Nov 03 12:24:17.497011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.496876    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf01078b-da80-420e-b7ff-8a6c1febca7e-istiod-ca-cert" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:17.497212 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.497040    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-data" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.497212 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.497093    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-certs" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.497392 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.497372    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.497435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.497399    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf01078b-da80-420e-b7ff-8a6c1febca7e-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.497435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.497414    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.497435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.497430    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.497540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.497445    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.498739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.498715    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-kube-api-access-lng2v" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "kube-api-access-lng2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:17.499063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.499046    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-token" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:17.499063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.499053    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-podinfo" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:24:17.499220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.499206    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-envoy" pod "bf01078b-da80-420e-b7ff-8a6c1febca7e" (UID: "bf01078b-da80-420e-b7ff-8a6c1febca7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.599156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.599123    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.599156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.599150    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lng2v\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-kube-api-access-lng2v\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.599156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.599160    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.599388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.599168    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf01078b-da80-420e-b7ff-8a6c1febca7e-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.615333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.615302    4909 generic.go:356] "Generic (PLEG): container finished" podID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerID="1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125" exitCode=0
Nov 03 12:24:17.615465 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.615367    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"
Nov 03 12:24:17.615465 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.615392    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" event={"ID":"bf01078b-da80-420e-b7ff-8a6c1febca7e","Type":"ContainerDied","Data":"1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125"}
Nov 03 12:24:17.615549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.615462    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r" event={"ID":"bf01078b-da80-420e-b7ff-8a6c1febca7e","Type":"ContainerDied","Data":"20947c8054ecc9bfc6b580cc660101e9d8f2d6003a109df023847e1c427d28a5"}
Nov 03 12:24:17.615549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.615498    4909 scope.go:122] "RemoveContainer" containerID="1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125"
Nov 03 12:24:17.625755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.625737    4909 scope.go:122] "RemoveContainer" containerID="1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125"
Nov 03 12:24:17.626032 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:17.626010    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125\": container with ID starting with 1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125 not found: ID does not exist" containerID="1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125"
Nov 03 12:24:17.626086 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.626040    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125"} err="failed to get container status \"1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125\": rpc error: code = NotFound desc = could not find container \"1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125\": container with ID starting with 1937b82f829dc885a8515790a18091a42f105907d87207f0b17b511ad720e125 not found: ID does not exist"
Nov 03 12:24:17.768384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.768309    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"]
Nov 03 12:24:17.773196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:17.773166    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-bmz4r"]
Nov 03 12:24:19.123959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:19.123929    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" path="/var/lib/kubelet/pods/bf01078b-da80-420e-b7ff-8a6c1febca7e/volumes"
Nov 03 12:24:19.601665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:19.601637    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:24:27.346426 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:27.346394    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"]
Nov 03 12:24:27.346849 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:27.346566    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy" containerID="cri-o://cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd" gracePeriod=30
Nov 03 12:24:27.677599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:27.677469    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:24:32.640319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.640294    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:24:32.710875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.710845    4909 generic.go:356] "Generic (PLEG): container finished" podID="64ca0240-f3e0-4357-8534-b48e322687a7" containerID="cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd" exitCode=0
Nov 03 12:24:32.711028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.710889    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"
Nov 03 12:24:32.711028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.710926    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" event={"ID":"64ca0240-f3e0-4357-8534-b48e322687a7","Type":"ContainerDied","Data":"cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd"}
Nov 03 12:24:32.711028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.711003    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w" event={"ID":"64ca0240-f3e0-4357-8534-b48e322687a7","Type":"ContainerDied","Data":"e10809903cc7390474c08c638cce3be034196a5245fad8bed81428092e1d66d5"}
Nov 03 12:24:32.711028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.711025    4909 scope.go:122] "RemoveContainer" containerID="cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd"
Nov 03 12:24:32.712167 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712148    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64ca0240-f3e0-4357-8534-b48e322687a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ca0240-f3e0-4357-8534-b48e322687a7-istio-podinfo\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712181    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-socket\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712265 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712251    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ca0240-f3e0-4357-8534-b48e322687a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ca0240-f3e0-4357-8534-b48e322687a7-istiod-ca-cert\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712365 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712279    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-istio-token\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-istio-token\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712365 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712335    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-credential-socket\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712434    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-envoy\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712478    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-data\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712523    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-certs\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712613 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712579    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-socket" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:32.712613 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712589    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-kube-api-access-tvj8x\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-kube-api-access-tvj8x\") pod \"64ca0240-f3e0-4357-8534-b48e322687a7\" (UID: \"64ca0240-f3e0-4357-8534-b48e322687a7\") "
Nov 03 12:24:32.712820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712657    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ca0240-f3e0-4357-8534-b48e322687a7-istiod-ca-cert" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:32.713224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.712999    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-credential-socket" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:32.713224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.713114    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-certs" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:32.713401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.713289    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.713401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.713313    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.713401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.713327    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.713401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.713341    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ca0240-f3e0-4357-8534-b48e322687a7-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.713634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.713400    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-data" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:32.714602 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.714575    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ca0240-f3e0-4357-8534-b48e322687a7-istio-podinfo" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:24:32.714759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.714598    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-istio-token" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:32.714967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.714947    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-kube-api-access-tvj8x" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "kube-api-access-tvj8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:32.715565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.715539    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-envoy" pod "64ca0240-f3e0-4357-8534-b48e322687a7" (UID: "64ca0240-f3e0-4357-8534-b48e322687a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:32.741158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.741138    4909 scope.go:122] "RemoveContainer" containerID="cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd"
Nov 03 12:24:32.741431 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:32.741410    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd\": container with ID starting with cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd not found: ID does not exist" containerID="cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd"
Nov 03 12:24:32.741476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.741441    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd"} err="failed to get container status \"cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd\": rpc error: code = NotFound desc = could not find container \"cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd\": container with ID starting with cd3f97aff8d3c150f392a5f0cdf78ac328c642be3a71f7ef6ed698e2e8790edd not found: ID does not exist"
Nov 03 12:24:32.849720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.844429    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.849720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.844464    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ca0240-f3e0-4357-8534-b48e322687a7-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.849720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.844477    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvj8x\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-kube-api-access-tvj8x\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.849720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.844491    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ca0240-f3e0-4357-8534-b48e322687a7-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:32.849720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:32.844502    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ca0240-f3e0-4357-8534-b48e322687a7-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:33.127591 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:33.127558    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"]
Nov 03 12:24:33.130230 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:33.130206    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-kwl4w"]
Nov 03 12:24:35.125836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:35.125807    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" path="/var/lib/kubelet/pods/64ca0240-f3e0-4357-8534-b48e322687a7/volumes"
Nov 03 12:24:39.404476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.404442    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"]
Nov 03 12:24:39.404914 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.404652    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" podUID="72662b2c-2f65-479e-8e81-a41ab7e24468" containerName="maas-api" containerID="cri-o://817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95" gracePeriod=30
Nov 03 12:24:39.564822 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:39.564780    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:24:39.739424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.739401    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:24:39.755913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.755886    4909 generic.go:356] "Generic (PLEG): container finished" podID="72662b2c-2f65-479e-8e81-a41ab7e24468" containerID="817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95" exitCode=0
Nov 03 12:24:39.756044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.755940    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"
Nov 03 12:24:39.756044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.755978    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" event={"ID":"72662b2c-2f65-479e-8e81-a41ab7e24468","Type":"ContainerDied","Data":"817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95"}
Nov 03 12:24:39.756132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.756042    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" event={"ID":"72662b2c-2f65-479e-8e81-a41ab7e24468","Type":"ContainerDied","Data":"ded420e025d4a20cb0687ce9cfc4034a70413c9c58cf4811f28887861b0185de"}
Nov 03 12:24:39.756132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.756071    4909 scope.go:122] "RemoveContainer" containerID="817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95"
Nov 03 12:24:39.765381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.765365    4909 scope.go:122] "RemoveContainer" containerID="817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95"
Nov 03 12:24:39.765637 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:24:39.765617    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95\": container with ID starting with 817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95 not found: ID does not exist" containerID="817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95"
Nov 03 12:24:39.765745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.765644    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95"} err="failed to get container status \"817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95\": rpc error: code = NotFound desc = could not find container \"817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95\": container with ID starting with 817e0351e554cfae4ba205e622aedb7e8464ae5d6529f50da0335d2f64016c95 not found: ID does not exist"
Nov 03 12:24:39.832459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.832438    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72662b2c-2f65-479e-8e81-a41ab7e24468-kube-api-access-j8rm7\" (UniqueName: \"kubernetes.io/projected/72662b2c-2f65-479e-8e81-a41ab7e24468-kube-api-access-j8rm7\") pod \"72662b2c-2f65-479e-8e81-a41ab7e24468\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") "
Nov 03 12:24:39.832563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.832506    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-metrics-certs\") pod \"72662b2c-2f65-479e-8e81-a41ab7e24468\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") "
Nov 03 12:24:39.832563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.832557    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls\") pod \"72662b2c-2f65-479e-8e81-a41ab7e24468\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") "
Nov 03 12:24:39.832680 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.832670    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72662b2c-2f65-479e-8e81-a41ab7e24468-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72662b2c-2f65-479e-8e81-a41ab7e24468-openshift-service-ca\") pod \"72662b2c-2f65-479e-8e81-a41ab7e24468\" (UID: \"72662b2c-2f65-479e-8e81-a41ab7e24468\") "
Nov 03 12:24:39.833041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.833016    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72662b2c-2f65-479e-8e81-a41ab7e24468-openshift-service-ca" pod "72662b2c-2f65-479e-8e81-a41ab7e24468" (UID: "72662b2c-2f65-479e-8e81-a41ab7e24468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:39.833189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.833170    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72662b2c-2f65-479e-8e81-a41ab7e24468-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:39.834579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.834558    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72662b2c-2f65-479e-8e81-a41ab7e24468-kube-api-access-j8rm7" pod "72662b2c-2f65-479e-8e81-a41ab7e24468" (UID: "72662b2c-2f65-479e-8e81-a41ab7e24468"). InnerVolumeSpecName "kube-api-access-j8rm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:39.834730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.834689    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-metrics-certs" pod "72662b2c-2f65-479e-8e81-a41ab7e24468" (UID: "72662b2c-2f65-479e-8e81-a41ab7e24468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:24:39.834938 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.834917    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls" pod "72662b2c-2f65-479e-8e81-a41ab7e24468" (UID: "72662b2c-2f65-479e-8e81-a41ab7e24468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:24:39.934013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.933981    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rm7\" (UniqueName: \"kubernetes.io/projected/72662b2c-2f65-479e-8e81-a41ab7e24468-kube-api-access-j8rm7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:39.934013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.934008    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:39.934013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:39.934019    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72662b2c-2f65-479e-8e81-a41ab7e24468-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:40.169237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.169210    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"]
Nov 03 12:24:40.171762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.171736    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd"]
Nov 03 12:24:40.529670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.529637    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"]
Nov 03 12:24:40.530131 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.529908    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7" podUID="1d6f7294-7021-48a8-a979-e447011bc9c1" containerName="maas-api" containerID="cri-o://1b320f25898bd52c6d0ff68165ec577ca99186abda64067fe20b1e46da790ec0" gracePeriod=30
Nov 03 12:24:40.765198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.765170    4909 generic.go:356] "Generic (PLEG): container finished" podID="1d6f7294-7021-48a8-a979-e447011bc9c1" containerID="1b320f25898bd52c6d0ff68165ec577ca99186abda64067fe20b1e46da790ec0" exitCode=0
Nov 03 12:24:40.765358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.765246    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7" event={"ID":"1d6f7294-7021-48a8-a979-e447011bc9c1","Type":"ContainerDied","Data":"1b320f25898bd52c6d0ff68165ec577ca99186abda64067fe20b1e46da790ec0"}
Nov 03 12:24:40.856126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.856096    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:24:40.957832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.957794    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6f7294-7021-48a8-a979-e447011bc9c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f7294-7021-48a8-a979-e447011bc9c1-openshift-service-ca\") pod \"1d6f7294-7021-48a8-a979-e447011bc9c1\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") "
Nov 03 12:24:41.007720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.981859    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-metrics-certs\") pod \"1d6f7294-7021-48a8-a979-e447011bc9c1\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") "
Nov 03 12:24:41.021950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.021915    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6f7294-7021-48a8-a979-e447011bc9c1-kube-api-access-nd2sv\" (UniqueName: \"kubernetes.io/projected/1d6f7294-7021-48a8-a979-e447011bc9c1-kube-api-access-nd2sv\") pod \"1d6f7294-7021-48a8-a979-e447011bc9c1\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") "
Nov 03 12:24:41.025568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.025538    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls\") pod \"1d6f7294-7021-48a8-a979-e447011bc9c1\" (UID: \"1d6f7294-7021-48a8-a979-e447011bc9c1\") "
Nov 03 12:24:41.037788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:40.976159    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6f7294-7021-48a8-a979-e447011bc9c1-openshift-service-ca" pod "1d6f7294-7021-48a8-a979-e447011bc9c1" (UID: "1d6f7294-7021-48a8-a979-e447011bc9c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:41.037788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.015823    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-metrics-certs" pod "1d6f7294-7021-48a8-a979-e447011bc9c1" (UID: "1d6f7294-7021-48a8-a979-e447011bc9c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:24:41.037788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.036903    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6f7294-7021-48a8-a979-e447011bc9c1-kube-api-access-nd2sv" pod "1d6f7294-7021-48a8-a979-e447011bc9c1" (UID: "1d6f7294-7021-48a8-a979-e447011bc9c1"). InnerVolumeSpecName "kube-api-access-nd2sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:41.037788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.037215    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls" pod "1d6f7294-7021-48a8-a979-e447011bc9c1" (UID: "1d6f7294-7021-48a8-a979-e447011bc9c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:24:41.123545 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.123514    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72662b2c-2f65-479e-8e81-a41ab7e24468" path="/var/lib/kubelet/pods/72662b2c-2f65-479e-8e81-a41ab7e24468/volumes"
Nov 03 12:24:41.127156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.127131    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.127288 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.127159    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2sv\" (UniqueName: \"kubernetes.io/projected/1d6f7294-7021-48a8-a979-e447011bc9c1-kube-api-access-nd2sv\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.127288 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.127174    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6f7294-7021-48a8-a979-e447011bc9c1-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.127288 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.127186    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f7294-7021-48a8-a979-e447011bc9c1-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.771625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.771593    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"
Nov 03 12:24:41.772093 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.771619    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7" event={"ID":"1d6f7294-7021-48a8-a979-e447011bc9c1","Type":"ContainerDied","Data":"c4aa8613f315f4ab30cd6c67c86ee4ea13dbb4da299c903cc2aa6ffdaa97a07a"}
Nov 03 12:24:41.772093 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.771680    4909 scope.go:122] "RemoveContainer" containerID="1b320f25898bd52c6d0ff68165ec577ca99186abda64067fe20b1e46da790ec0"
Nov 03 12:24:41.926032 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.925963    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"]
Nov 03 12:24:41.972087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:41.972062    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-583bcf-b49ffbf96-78fg7"]
Nov 03 12:24:42.619195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:42.619159    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-059838-5d5f9c856-vdhfd" podUID="72662b2c-2f65-479e-8e81-a41ab7e24468" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.105:8443/health\": context deadline exceeded"
Nov 03 12:24:43.123834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:43.123804    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6f7294-7021-48a8-a979-e447011bc9c1" path="/var/lib/kubelet/pods/1d6f7294-7021-48a8-a979-e447011bc9c1/volumes"
Nov 03 12:24:50.260351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:50.260306    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 12:24:50.277966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:50.277922    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:24:50.359821 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:24:50.359780    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:25:08.055045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055010    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"]
Nov 03 12:25:08.055556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055537    4909 cpu_manager.go:414] "containerMap: removing container" podUID="72662b2c-2f65-479e-8e81-a41ab7e24468" containerName="maas-api"
Nov 03 12:25:08.055556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055555    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72662b2c-2f65-479e-8e81-a41ab7e24468" containerName="maas-api"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055566    4909 cpu_manager.go:414] "containerMap: removing container" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055573    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055579    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055584    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055592    4909 cpu_manager.go:414] "containerMap: removing container" podUID="1d6f7294-7021-48a8-a979-e447011bc9c1" containerName="maas-api"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055596    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6f7294-7021-48a8-a979-e447011bc9c1" containerName="maas-api"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055619    4909 cpu_manager.go:414] "containerMap: removing container" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055624    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy"
Nov 03 12:25:08.055733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055721    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf01078b-da80-420e-b7ff-8a6c1febca7e" containerName="istio-proxy"
Nov 03 12:25:08.056077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055771    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="72662b2c-2f65-479e-8e81-a41ab7e24468" containerName="maas-api"
Nov 03 12:25:08.056077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055784    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ca0240-f3e0-4357-8534-b48e322687a7" containerName="istio-proxy"
Nov 03 12:25:08.056077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055795    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b851622c-afe9-4f41-9c18-0e04404289ed" containerName="istio-proxy"
Nov 03 12:25:08.056077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.055835    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6f7294-7021-48a8-a979-e447011bc9c1" containerName="maas-api"
Nov 03 12:25:08.061078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.061061    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.143595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.143564    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-980a31-openshift-default-dockercfg-67kh2\""
Nov 03 12:25:08.158118 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.158082    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"]
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.161681    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-token\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.161919    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-certs\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.161957    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-socket\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.161986    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-podinfo\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.162014    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-credential-socket\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.162246    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dc2x\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-kube-api-access-8dc2x\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.162361    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-data\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.162402    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7264bb7f-0dac-420c-8201-37a8af95e77b-istiod-ca-cert\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.166071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.162435    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-envoy\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.327834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.316686    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-data\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.328080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.328061    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7264bb7f-0dac-420c-8201-37a8af95e77b-istiod-ca-cert\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.328190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.328180    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-envoy\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.328341 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.328330    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-token\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.348962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.348927    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-certs\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.349218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.349190    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-socket\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.349336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.349322    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-podinfo\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.349444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.349430    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-credential-socket\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.349566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.349551    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dc2x\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-kube-api-access-8dc2x\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.357721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.329025    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-data\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.360829 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.351649    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-socket\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.361296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.351900    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-certs\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.361490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.329512    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7264bb7f-0dac-420c-8201-37a8af95e77b-istiod-ca-cert\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.361616 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.352615    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-credential-socket\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.361752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.357523    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-envoy\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.363223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.363197    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-podinfo\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.397308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.397257    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-token\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.412203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.412163    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dc2x\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-kube-api-access-8dc2x\") pod \"e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") " pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:08.675060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:08.674976    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:09.090344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:09.090315    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"]
Nov 03 12:25:09.091637 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:25:09.091611    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7264bb7f_0dac_420c_8201_37a8af95e77b.slice/crio-d0315a1f637ef221e85f7e3ce0f6fb6b07ccbfc31dddc02b18edf5b5f01f7d3d WatchSource:0}: Error finding container d0315a1f637ef221e85f7e3ce0f6fb6b07ccbfc31dddc02b18edf5b5f01f7d3d: Status 404 returned error can't find the container with id d0315a1f637ef221e85f7e3ce0f6fb6b07ccbfc31dddc02b18edf5b5f01f7d3d
Nov 03 12:25:09.093865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:09.093835    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:25:09.093957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:09.093891    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:25:09.093957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:09.093922    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:25:09.563580 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:25:09.563538    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:25:10.014677 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:10.014643    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" event={"ID":"7264bb7f-0dac-420c-8201-37a8af95e77b","Type":"ContainerStarted","Data":"cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70"}
Nov 03 12:25:10.014916 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:10.014692    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" event={"ID":"7264bb7f-0dac-420c-8201-37a8af95e77b","Type":"ContainerStarted","Data":"d0315a1f637ef221e85f7e3ce0f6fb6b07ccbfc31dddc02b18edf5b5f01f7d3d"}
Nov 03 12:25:10.169526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:10.169472    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podStartSLOduration=2.169348621 podStartE2EDuration="2.169348621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:10.168260058 +0000 UTC m=+2059.180726718" watchObservedRunningTime="2026-11-03 12:25:10.169348621 +0000 UTC m=+2059.181815285"
Nov 03 12:25:10.676078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:10.676046    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:10.689770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:10.689741    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:11.022363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:11.022332    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:11.023258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:11.023222    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:25:12.379202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:12.379162    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:25:35.277534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:35.277455    4909 generic.go:356] "Generic (PLEG): container finished" podID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerID="d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f" exitCode=0
Nov 03 12:25:35.278055 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:35.277536    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" event={"ID":"f3e6b8e3-35ed-4714-b43e-a63d7267daa2","Type":"ContainerDied","Data":"d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f"}
Nov 03 12:25:35.278055 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:35.277908    4909 scope.go:122] "RemoveContainer" containerID="d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f"
Nov 03 12:25:35.279281 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:35.279256    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:25:35.279357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:35.279322    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:25:35.279409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:35.279358    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:25:36.285520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:36.285486    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" event={"ID":"f3e6b8e3-35ed-4714-b43e-a63d7267daa2","Type":"ContainerStarted","Data":"bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f"}
Nov 03 12:25:37.222190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:37.222151    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:25:37.223548 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:37.223521    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:25:37.223683 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:37.223567    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 12:25:38.221755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:38.221724    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:25:38.222132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:38.221771    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 12:25:39.222272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:39.222236    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:25:39.222756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:39.222296    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 12:25:39.587261 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:25:39.587175    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:25:40.221497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:40.221466    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:25:40.221670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:40.221521    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 12:25:40.319172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:40.319141    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:25:41.221962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.221929    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:25:41.222132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.221996    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 12:25:41.468678 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.468641    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"]
Nov 03 12:25:41.473069 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.473016    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.526319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.526276    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"]
Nov 03 12:25:41.568882 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.568845    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b6db28-serving-cert\""
Nov 03 12:25:41.589076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.589031    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.589245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.589084    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-metrics-certs\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.589245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.589221    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52mg\" (UniqueName: \"kubernetes.io/projected/39c4316d-2981-4749-b91e-e27693344729-kube-api-access-t52mg\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.589469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.589451    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4316d-2981-4749-b91e-e27693344729-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.690797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.690766    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4316d-2981-4749-b91e-e27693344729-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.690975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.690885    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.690975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.690912    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-metrics-certs\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.690975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.690949    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52mg\" (UniqueName: \"kubernetes.io/projected/39c4316d-2981-4749-b91e-e27693344729-kube-api-access-t52mg\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.691114 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:25:41.691039    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-serving-cert: secret "maas-api-e2e-praxis-a-b6db28-serving-cert" not found
Nov 03 12:25:41.691171 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:25:41.691119    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls podName:39c4316d-2981-4749-b91e-e27693344729 nodeName:}" failed. No retries permitted until 2026-11-03 12:25:42.191098653 +0000 UTC m=+2091.203565294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls") pod "maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299" (UID: "39c4316d-2981-4749-b91e-e27693344729") : secret "maas-api-e2e-praxis-a-b6db28-serving-cert" not found
Nov 03 12:25:41.691543 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.691523    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4316d-2981-4749-b91e-e27693344729-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.693587 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.693559    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-metrics-certs\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:41.783447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:41.783365    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52mg\" (UniqueName: \"kubernetes.io/projected/39c4316d-2981-4749-b91e-e27693344729-kube-api-access-t52mg\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:42.200068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:42.200038    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:42.202643 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:42.202619    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:42.226069 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:42.226044    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:25:42.226846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:42.226828    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:25:42.387736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:42.387694    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:42.754744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:42.754712    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"]
Nov 03 12:25:42.760513 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:25:42.760488    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c4316d_2981_4749_b91e_e27693344729.slice/crio-01fbc59925f9f0888e7266c2e79846b4512b87858d5558c5c6f17cde22be2c50 WatchSource:0}: Error finding container 01fbc59925f9f0888e7266c2e79846b4512b87858d5558c5c6f17cde22be2c50: Status 404 returned error can't find the container with id 01fbc59925f9f0888e7266c2e79846b4512b87858d5558c5c6f17cde22be2c50
Nov 03 12:25:42.761925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:42.761906    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:25:43.325725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:43.325671    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299" event={"ID":"39c4316d-2981-4749-b91e-e27693344729","Type":"ContainerStarted","Data":"01fbc59925f9f0888e7266c2e79846b4512b87858d5558c5c6f17cde22be2c50"}
Nov 03 12:25:44.333048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:44.333005    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299" event={"ID":"39c4316d-2981-4749-b91e-e27693344729","Type":"ContainerStarted","Data":"f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a"}
Nov 03 12:25:44.333483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:44.333078    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:44.439830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:44.439779    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299" podStartSLOduration=2.5227195460000003 podStartE2EDuration="3.439764968s" totalImagesPullingTime="917.045422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:44.438536542 +0000 UTC m=+2093.451003206" watchObservedRunningTime="2026-11-03 12:25:44.439764968 +0000 UTC m=+2093.452231633"
Nov 03 12:25:50.344187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:50.344158    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:25:51.382040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:51.382005    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:25:51.383840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:51.383819    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:25:57.883230 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:25:57.883188    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:26:09.566638 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:26:09.566548    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:26:24.235329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:24.235297    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:26:34.630651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:34.630618    4909 generic.go:356] "Generic (PLEG): container finished" podID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerID="cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70" exitCode=0
Nov 03 12:26:34.631115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:34.630688    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" event={"ID":"7264bb7f-0dac-420c-8201-37a8af95e77b","Type":"ContainerDied","Data":"cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70"}
Nov 03 12:26:34.631115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:34.631081    4909 scope.go:122] "RemoveContainer" containerID="cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70"
Nov 03 12:26:34.632666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:34.632640    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:26:34.632772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:34.632727    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:26:34.632772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:34.632761    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:26:35.637825 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:35.637787    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" event={"ID":"7264bb7f-0dac-420c-8201-37a8af95e77b","Type":"ContainerStarted","Data":"f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3"}
Nov 03 12:26:35.675792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:35.675752    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:26:35.676000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:35.675973    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:26:35.676077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:35.676029    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 12:26:36.675771 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:36.675736    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:26:36.676151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:36.675805    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 12:26:37.675910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:37.675871    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:26:37.676392 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:37.675932    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 12:26:38.675323 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:38.675289    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:26:38.675509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:38.675349    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 12:26:39.586908 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:26:39.586869    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:26:39.675667 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:39.675637    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:26:39.675850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:39.675726    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 12:26:40.675651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:40.675615    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:26:40.676154 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:40.675681    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 12:26:41.022970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:41.022939    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:26:41.680828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:41.680790    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:26:41.695621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:41.695592    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:26:49.102749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.102717    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"]
Nov 03 12:26:49.107284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.107262    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.176722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.176371    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-40ffcd-openshift-default-dockercfg-4tws4\""
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266408    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-credential-socket\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266487    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-token\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266548    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzmjm\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-kube-api-access-jzmjm\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266590    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-socket\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266681    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istiod-ca-cert\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266753    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-data\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266789    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-podinfo\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266818    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-certs\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.271724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.266932    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-envoy\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.294110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.291626    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"]
Nov 03 12:26:49.383511 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.383425    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-envoy\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.399541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.399500    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-credential-socket\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.399783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.399518    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-credential-socket\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.399930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.399913    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-token\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.400068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.400054    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzmjm\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-kube-api-access-jzmjm\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.400178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.400166    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-socket\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.400346 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.400333    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istiod-ca-cert\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.400468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.400456    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-data\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.400586 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.400572    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-podinfo\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.400684 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.400672    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-certs\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.401124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.401097    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-socket\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.401311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.401284    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-envoy\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.401442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.401423    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-certs\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.401593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.401571    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-data\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.402295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.402274    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istiod-ca-cert\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.406357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.406338    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-podinfo\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.453751 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.453724    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-token\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.453864 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.453802    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzmjm\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-kube-api-access-jzmjm\") pod \"e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") " pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:49.721053 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:49.721021    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:50.133686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.133607    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"]
Nov 03 12:26:50.140031 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.140012    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.140412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.140386    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"]
Nov 03 12:26:50.141971 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:26:50.141946    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91fdb8b_3546_4258_9068_74b6b5a7c2d8.slice/crio-df76582eea9c57ff8cbc497afb646fc5179994408168e57b1e27cb9c06c52efd WatchSource:0}: Error finding container df76582eea9c57ff8cbc497afb646fc5179994408168e57b1e27cb9c06c52efd: Status 404 returned error can't find the container with id df76582eea9c57ff8cbc497afb646fc5179994408168e57b1e27cb9c06c52efd
Nov 03 12:26:50.144364 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.144333    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:26:50.144463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.144388    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:26:50.144463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.144415    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:26:50.189370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.189343    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2f3eb2\"/\"default-dockercfg-gn7bm\""
Nov 03 12:26:50.189568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.189539    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2f3eb2\"/\"ipp-route-2f3eb2-kserve-self-signed-certs\""
Nov 03 12:26:50.189646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.189633    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2f3eb2\"/\"kube-root-ca.crt\""
Nov 03 12:26:50.189775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.189759    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2f3eb2\"/\"openshift-service-ca.crt\""
Nov 03 12:26:50.192554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.192529    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"]
Nov 03 12:26:50.324727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.318220    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83aab95e-fca5-4637-a151-85bd7566b3c5-tls-certs\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.324727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.318357    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29p9j\" (UniqueName: \"kubernetes.io/projected/83aab95e-fca5-4637-a151-85bd7566b3c5-kube-api-access-29p9j\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.324727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.318395    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-dshm\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.324727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.318496    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-model-cache\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.324727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.318591    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-home\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.324727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.318641    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-tmp-dir\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.419929    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29p9j\" (UniqueName: \"kubernetes.io/projected/83aab95e-fca5-4637-a151-85bd7566b3c5-kube-api-access-29p9j\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.419976    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-dshm\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.420081    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-model-cache\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.420171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-home\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420352 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.420225    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-tmp-dir\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420352 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.420314    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83aab95e-fca5-4637-a151-85bd7566b3c5-tls-certs\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.420478    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-model-cache\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.420580    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-home\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.420656 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.420603    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-tmp-dir\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.422399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.422374    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-dshm\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.422906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.422881    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83aab95e-fca5-4637-a151-85bd7566b3c5-tls-certs\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.473394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.473365    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29p9j\" (UniqueName: \"kubernetes.io/projected/83aab95e-fca5-4637-a151-85bd7566b3c5-kube-api-access-29p9j\") pod \"ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") " pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.753499 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.753380    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:50.755972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.755940    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" event={"ID":"c91fdb8b-3546-4258-9068-74b6b5a7c2d8","Type":"ContainerStarted","Data":"04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058"}
Nov 03 12:26:50.756082 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.756000    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" event={"ID":"c91fdb8b-3546-4258-9068-74b6b5a7c2d8","Type":"ContainerStarted","Data":"df76582eea9c57ff8cbc497afb646fc5179994408168e57b1e27cb9c06c52efd"}
Nov 03 12:26:50.865161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:50.865113    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podStartSLOduration=1.865099289 podStartE2EDuration="1.865099289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:50.864680938 +0000 UTC m=+2159.877147602" watchObservedRunningTime="2026-11-03 12:26:50.865099289 +0000 UTC m=+2159.877566015"
Nov 03 12:26:51.142710 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:51.142673    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"]
Nov 03 12:26:51.144575 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:26:51.144551    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83aab95e_fca5_4637_a151_85bd7566b3c5.slice/crio-2b7372755edcceefe55a49eab840a354edc0d42ed8c308421d166c9091328987 WatchSource:0}: Error finding container 2b7372755edcceefe55a49eab840a354edc0d42ed8c308421d166c9091328987: Status 404 returned error can't find the container with id 2b7372755edcceefe55a49eab840a354edc0d42ed8c308421d166c9091328987
Nov 03 12:26:51.722075 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:51.722036    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:51.729444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:51.729422    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:51.765833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:51.765795    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj" event={"ID":"83aab95e-fca5-4637-a151-85bd7566b3c5","Type":"ContainerStarted","Data":"3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520"}
Nov 03 12:26:51.765975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:51.765845    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj" event={"ID":"83aab95e-fca5-4637-a151-85bd7566b3c5","Type":"ContainerStarted","Data":"2b7372755edcceefe55a49eab840a354edc0d42ed8c308421d166c9091328987"}
Nov 03 12:26:51.766092 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:51.766068    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:51.767150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:51.767125    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:26:52.058464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:52.058345    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj" podStartSLOduration=2.058318333 podStartE2EDuration="2.058318333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:52.050920323 +0000 UTC m=+2161.063386986" watchObservedRunningTime="2026-11-03 12:26:52.058318333 +0000 UTC m=+2161.070784999"
Nov 03 12:26:52.754410 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:52.754381    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:26:57.084281 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.084251    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"]
Nov 03 12:26:57.088519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.088498    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.187798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.186068    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-40ffcd-dockercfg-dmvjr\""
Nov 03 12:26:57.191719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.188822    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-40ffcd\""
Nov 03 12:26:57.196715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.194114    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"]
Nov 03 12:26:57.245622 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.245583    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjcth\" (UniqueName: \"kubernetes.io/projected/ae043112-933c-4a53-bea8-292b0d08d034-kube-api-access-kjcth\") pod \"payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.246381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.246339    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae043112-933c-4a53-bea8-292b0d08d034-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.279625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.279585    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"]
Nov 03 12:26:57.309967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.309940    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:57.346656 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.346531    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"]
Nov 03 12:26:57.382041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.364522    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e0649-90d2-4a95-9e58-a48c24514abf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:57.382464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.382427    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjcth\" (UniqueName: \"kubernetes.io/projected/ae043112-933c-4a53-bea8-292b0d08d034-kube-api-access-kjcth\") pod \"payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.392858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.388663    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gwl\" (UniqueName: \"kubernetes.io/projected/763e0649-90d2-4a95-9e58-a48c24514abf-kube-api-access-g9gwl\") pod \"payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:57.393227 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.393200    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae043112-933c-4a53-bea8-292b0d08d034-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.395232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.395207    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae043112-933c-4a53-bea8-292b0d08d034-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.448207 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.448178    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjcth\" (UniqueName: \"kubernetes.io/projected/ae043112-933c-4a53-bea8-292b0d08d034-kube-api-access-kjcth\") pod \"payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.495202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.495174    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e0649-90d2-4a95-9e58-a48c24514abf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:57.495359 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.495237    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gwl\" (UniqueName: \"kubernetes.io/projected/763e0649-90d2-4a95-9e58-a48c24514abf-kube-api-access-g9gwl\") pod \"payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:57.495850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.495830    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e0649-90d2-4a95-9e58-a48c24514abf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:57.547435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.547409    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gwl\" (UniqueName: \"kubernetes.io/projected/763e0649-90d2-4a95-9e58-a48c24514abf-kube-api-access-g9gwl\") pod \"payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:57.703088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.703057    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:57.726941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:57.726900    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:58.077326 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.077302    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"]
Nov 03 12:26:58.078882 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:26:58.078852    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae043112_933c_4a53_bea8_292b0d08d034.slice/crio-dad80936b6b6cdd6e81beba1c78b63d5c9d4a5bf536ec5836af687e70d372dc7 WatchSource:0}: Error finding container dad80936b6b6cdd6e81beba1c78b63d5c9d4a5bf536ec5836af687e70d372dc7: Status 404 returned error can't find the container with id dad80936b6b6cdd6e81beba1c78b63d5c9d4a5bf536ec5836af687e70d372dc7
Nov 03 12:26:58.125725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.125685    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"]
Nov 03 12:26:58.143772 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:26:58.143749    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763e0649_90d2_4a95_9e58_a48c24514abf.slice/crio-9fa81257418ab7a443f3c760f9e61116993d7a3129287aec29da518a69fc9611 WatchSource:0}: Error finding container 9fa81257418ab7a443f3c760f9e61116993d7a3129287aec29da518a69fc9611: Status 404 returned error can't find the container with id 9fa81257418ab7a443f3c760f9e61116993d7a3129287aec29da518a69fc9611
Nov 03 12:26:58.820128 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.820088    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f" event={"ID":"763e0649-90d2-4a95-9e58-a48c24514abf","Type":"ContainerStarted","Data":"5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc"}
Nov 03 12:26:58.820336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.820244    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:26:58.820336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.820290    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f" event={"ID":"763e0649-90d2-4a95-9e58-a48c24514abf","Type":"ContainerStarted","Data":"9fa81257418ab7a443f3c760f9e61116993d7a3129287aec29da518a69fc9611"}
Nov 03 12:26:58.821727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.821689    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z" event={"ID":"ae043112-933c-4a53-bea8-292b0d08d034","Type":"ContainerStarted","Data":"00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997"}
Nov 03 12:26:58.821835 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.821740    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z" event={"ID":"ae043112-933c-4a53-bea8-292b0d08d034","Type":"ContainerStarted","Data":"dad80936b6b6cdd6e81beba1c78b63d5c9d4a5bf536ec5836af687e70d372dc7"}
Nov 03 12:26:58.821835 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.821785    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:26:58.926198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:58.926145    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f" podStartSLOduration=1.926132629 podStartE2EDuration="1.926132629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:58.924038976 +0000 UTC m=+2167.936505638" watchObservedRunningTime="2026-11-03 12:26:58.926132629 +0000 UTC m=+2167.938599291"
Nov 03 12:26:59.052389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:26:59.051515    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z" podStartSLOduration=2.05149266 podStartE2EDuration="2.05149266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:59.029007576 +0000 UTC m=+2168.041474239" watchObservedRunningTime="2026-11-03 12:26:59.05149266 +0000 UTC m=+2168.063959324"
Nov 03 12:27:00.754603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:00.754570    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:27:00.767236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:00.767214    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:27:00.845673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:00.845647    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:27:07.486715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:07.486677    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:27:09.564578 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:27:09.564543    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:27:09.830900 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:09.830815    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:27:09.830900 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:09.830889    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:27:18.880824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:18.880793    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"]
Nov 03 12:27:23.881879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:23.881831    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg" podUID="2f977abf-28a1-4ba9-8d00-9845c7671b92" containerName="payload-processing" containerID="cri-o://5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f" gracePeriod=25
Nov 03 12:27:24.151073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.151007    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:27:24.185846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.185812    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f977abf-28a1-4ba9-8d00-9845c7671b92-kube-api-access-l9lkv\" (UniqueName: \"kubernetes.io/projected/2f977abf-28a1-4ba9-8d00-9845c7671b92-kube-api-access-l9lkv\") pod \"2f977abf-28a1-4ba9-8d00-9845c7671b92\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") "
Nov 03 12:27:24.185983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.185886    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f977abf-28a1-4ba9-8d00-9845c7671b92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f977abf-28a1-4ba9-8d00-9845c7671b92-plugins-config-volume\") pod \"2f977abf-28a1-4ba9-8d00-9845c7671b92\" (UID: \"2f977abf-28a1-4ba9-8d00-9845c7671b92\") "
Nov 03 12:27:24.186257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.186229    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f977abf-28a1-4ba9-8d00-9845c7671b92-plugins-config-volume" pod "2f977abf-28a1-4ba9-8d00-9845c7671b92" (UID: "2f977abf-28a1-4ba9-8d00-9845c7671b92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:27:24.186920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.186893    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f977abf-28a1-4ba9-8d00-9845c7671b92-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:24.188037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.188019    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f977abf-28a1-4ba9-8d00-9845c7671b92-kube-api-access-l9lkv" pod "2f977abf-28a1-4ba9-8d00-9845c7671b92" (UID: "2f977abf-28a1-4ba9-8d00-9845c7671b92"). InnerVolumeSpecName "kube-api-access-l9lkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:27:24.287707 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.287674    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9lkv\" (UniqueName: \"kubernetes.io/projected/2f977abf-28a1-4ba9-8d00-9845c7671b92-kube-api-access-l9lkv\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:24.978763 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.978694    4909 generic.go:356] "Generic (PLEG): container finished" podID="2f977abf-28a1-4ba9-8d00-9845c7671b92" containerID="5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f" exitCode=0
Nov 03 12:27:24.979159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.978771    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"
Nov 03 12:27:24.979159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.978793    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg" event={"ID":"2f977abf-28a1-4ba9-8d00-9845c7671b92","Type":"ContainerDied","Data":"5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f"}
Nov 03 12:27:24.979159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.978878    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg" event={"ID":"2f977abf-28a1-4ba9-8d00-9845c7671b92","Type":"ContainerDied","Data":"a5203247c16062a59c0b70e158ac6e00fb4954587baeab7d408f9af2a633d511"}
Nov 03 12:27:24.979159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.978902    4909 scope.go:122] "RemoveContainer" containerID="5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f"
Nov 03 12:27:24.988615 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.988596    4909 scope.go:122] "RemoveContainer" containerID="5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f"
Nov 03 12:27:24.988908 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:27:24.988889    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f\": container with ID starting with 5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f not found: ID does not exist" containerID="5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f"
Nov 03 12:27:24.988977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:24.988916    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f"} err="failed to get container status \"5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f\": rpc error: code = NotFound desc = could not find container \"5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f\": container with ID starting with 5a36db8a358b96b90af45e25367f6da0b300b62672101124d7722780de39488f not found: ID does not exist"
Nov 03 12:27:25.131632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:25.131601    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"]
Nov 03 12:27:25.136412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:25.136378    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c8ee66-6c7bbbdfbf-ztwlg"]
Nov 03 12:27:27.124905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:27.124872    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f977abf-28a1-4ba9-8d00-9845c7671b92" path="/var/lib/kubelet/pods/2f977abf-28a1-4ba9-8d00-9845c7671b92/volumes"
Nov 03 12:27:39.564815 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:27:39.564737    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:27:50.925177 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:50.925139    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:27:52.601141 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:52.601108    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"]
Nov 03 12:27:52.601539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:52.601328    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm" podUID="c658c628-1456-4e92-a3aa-12a12832e307" containerName="maas-api" containerID="cri-o://1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9" gracePeriod=30
Nov 03 12:27:52.930547 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:52.930521    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:27:53.003816 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.003766    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c658c628-1456-4e92-a3aa-12a12832e307-kube-api-access-86mpr\" (UniqueName: \"kubernetes.io/projected/c658c628-1456-4e92-a3aa-12a12832e307-kube-api-access-86mpr\") pod \"c658c628-1456-4e92-a3aa-12a12832e307\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") "
Nov 03 12:27:53.004156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.004131    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls\") pod \"c658c628-1456-4e92-a3aa-12a12832e307\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") "
Nov 03 12:27:53.004399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.004380    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-metrics-certs\") pod \"c658c628-1456-4e92-a3aa-12a12832e307\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") "
Nov 03 12:27:53.004635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.004613    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c658c628-1456-4e92-a3aa-12a12832e307-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c658c628-1456-4e92-a3aa-12a12832e307-openshift-service-ca\") pod \"c658c628-1456-4e92-a3aa-12a12832e307\" (UID: \"c658c628-1456-4e92-a3aa-12a12832e307\") "
Nov 03 12:27:53.005766 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.005744    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c658c628-1456-4e92-a3aa-12a12832e307-openshift-service-ca" pod "c658c628-1456-4e92-a3aa-12a12832e307" (UID: "c658c628-1456-4e92-a3aa-12a12832e307"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:27:53.014185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.014154    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-metrics-certs" pod "c658c628-1456-4e92-a3aa-12a12832e307" (UID: "c658c628-1456-4e92-a3aa-12a12832e307"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:27:53.037970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.037900    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c658c628-1456-4e92-a3aa-12a12832e307-kube-api-access-86mpr" pod "c658c628-1456-4e92-a3aa-12a12832e307" (UID: "c658c628-1456-4e92-a3aa-12a12832e307"). InnerVolumeSpecName "kube-api-access-86mpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:27:53.038853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.038820    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls" pod "c658c628-1456-4e92-a3aa-12a12832e307" (UID: "c658c628-1456-4e92-a3aa-12a12832e307"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:27:53.106252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.106221    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c658c628-1456-4e92-a3aa-12a12832e307-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:53.106252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.106249    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86mpr\" (UniqueName: \"kubernetes.io/projected/c658c628-1456-4e92-a3aa-12a12832e307-kube-api-access-86mpr\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:53.106252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.106259    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:53.106481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.106268    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c658c628-1456-4e92-a3aa-12a12832e307-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:53.146143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.146111    4909 generic.go:356] "Generic (PLEG): container finished" podID="c658c628-1456-4e92-a3aa-12a12832e307" containerID="1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9" exitCode=0
Nov 03 12:27:53.146277 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.146155    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"
Nov 03 12:27:53.146277 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.146196    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm" event={"ID":"c658c628-1456-4e92-a3aa-12a12832e307","Type":"ContainerDied","Data":"1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9"}
Nov 03 12:27:53.146277 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.146249    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm" event={"ID":"c658c628-1456-4e92-a3aa-12a12832e307","Type":"ContainerDied","Data":"f1ef08a6dab2c0c05fd1dc5ca1db87b82373a7c407850af17fee6653273416dd"}
Nov 03 12:27:53.146277 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.146271    4909 scope.go:122] "RemoveContainer" containerID="1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9"
Nov 03 12:27:53.155534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.155516    4909 scope.go:122] "RemoveContainer" containerID="1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9"
Nov 03 12:27:53.155801 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:27:53.155780    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9\": container with ID starting with 1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9 not found: ID does not exist" containerID="1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9"
Nov 03 12:27:53.155857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.155809    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9"} err="failed to get container status \"1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9\": rpc error: code = NotFound desc = could not find container \"1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9\": container with ID starting with 1959cc7e1f963438f726e450a2750daad592132ff565579c7d9bf8d1b0ed85e9 not found: ID does not exist"
Nov 03 12:27:53.298452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.298395    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"]
Nov 03 12:27:53.301213 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:53.301193    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c8ee66-758c766745-nw6lm"]
Nov 03 12:27:55.123425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:27:55.123393    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c658c628-1456-4e92-a3aa-12a12832e307" path="/var/lib/kubelet/pods/c658c628-1456-4e92-a3aa-12a12832e307/volumes"
Nov 03 12:28:03.564572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.564537    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"]
Nov 03 12:28:03.565090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.565076    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c658c628-1456-4e92-a3aa-12a12832e307" containerName="maas-api"
Nov 03 12:28:03.565131 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.565091    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c658c628-1456-4e92-a3aa-12a12832e307" containerName="maas-api"
Nov 03 12:28:03.565131 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.565122    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2f977abf-28a1-4ba9-8d00-9845c7671b92" containerName="payload-processing"
Nov 03 12:28:03.565131 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.565128    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f977abf-28a1-4ba9-8d00-9845c7671b92" containerName="payload-processing"
Nov 03 12:28:03.565224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.565212    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f977abf-28a1-4ba9-8d00-9845c7671b92" containerName="payload-processing"
Nov 03 12:28:03.565258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.565224    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c658c628-1456-4e92-a3aa-12a12832e307" containerName="maas-api"
Nov 03 12:28:03.569772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.569754    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.652184 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.652145    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.652338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.652221    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b92641-53e0-43b5-9e0b-6de2831d67d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.652420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.652389    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-metrics-certs\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.652518 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.652497    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl687\" (UniqueName: \"kubernetes.io/projected/d2b92641-53e0-43b5-9e0b-6de2831d67d7-kube-api-access-jl687\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.665079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.665053    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-04d73d-serving-cert\""
Nov 03 12:28:03.668013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.667993    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"]
Nov 03 12:28:03.754148 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.754117    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-metrics-certs\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.754296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.754176    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl687\" (UniqueName: \"kubernetes.io/projected/d2b92641-53e0-43b5-9e0b-6de2831d67d7-kube-api-access-jl687\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.754296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.754286    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.754377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.754315    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b92641-53e0-43b5-9e0b-6de2831d67d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.754447 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:28:03.754427    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-serving-cert: secret "maas-api-e2e-praxis-b-04d73d-serving-cert" not found
Nov 03 12:28:03.754522 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:28:03.754511    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls podName:d2b92641-53e0-43b5-9e0b-6de2831d67d7 nodeName:}" failed. No retries permitted until 2026-11-03 12:28:04.254489662 +0000 UTC m=+2233.266956303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls") pod "maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q" (UID: "d2b92641-53e0-43b5-9e0b-6de2831d67d7") : secret "maas-api-e2e-praxis-b-04d73d-serving-cert" not found
Nov 03 12:28:03.754965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.754931    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b92641-53e0-43b5-9e0b-6de2831d67d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.756857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.756836    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-metrics-certs\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:03.853773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:03.848470    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl687\" (UniqueName: \"kubernetes.io/projected/d2b92641-53e0-43b5-9e0b-6de2831d67d7-kube-api-access-jl687\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:04.265873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:04.265842    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:04.268292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:04.268268    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:04.483313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:04.483284    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:04.851552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:04.851517    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"]
Nov 03 12:28:04.853521 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:28:04.853490    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b92641_53e0_43b5_9e0b_6de2831d67d7.slice/crio-ed719293f47a037d2d55067c3b60c5094fdeb0cdedaa5c8304390b44c9732e69 WatchSource:0}: Error finding container ed719293f47a037d2d55067c3b60c5094fdeb0cdedaa5c8304390b44c9732e69: Status 404 returned error can't find the container with id ed719293f47a037d2d55067c3b60c5094fdeb0cdedaa5c8304390b44c9732e69
Nov 03 12:28:05.222322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:05.222286    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q" event={"ID":"d2b92641-53e0-43b5-9e0b-6de2831d67d7","Type":"ContainerStarted","Data":"ed719293f47a037d2d55067c3b60c5094fdeb0cdedaa5c8304390b44c9732e69"}
Nov 03 12:28:06.230657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:06.230626    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q" event={"ID":"d2b92641-53e0-43b5-9e0b-6de2831d67d7","Type":"ContainerStarted","Data":"690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7"}
Nov 03 12:28:06.231131 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:06.230694    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:06.337950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:06.337891    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q" podStartSLOduration=2.318409549 podStartE2EDuration="3.337878195s" totalImagesPullingTime="1.019468646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:28:06.33551288 +0000 UTC m=+2235.347979543" watchObservedRunningTime="2026-11-03 12:28:06.337878195 +0000 UTC m=+2235.350344857"
Nov 03 12:28:09.564638 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:28:09.564599    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:28:12.241309 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:12.241278    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:28:18.699354 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:18.699319    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"]
Nov 03 12:28:18.699828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:18.699528    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy" containerID="cri-o://f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3" gracePeriod=30
Nov 03 12:28:23.963643 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:23.963581    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:28:24.034075 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034046    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-credential-socket\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034125    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7264bb7f-0dac-420c-8201-37a8af95e77b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7264bb7f-0dac-420c-8201-37a8af95e77b-istiod-ca-cert\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034179    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-certs\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034201    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-token\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-token\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034238    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-envoy\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034359    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-credential-socket" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:24.034472 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034389    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-podinfo\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034472 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034421    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-socket\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034477    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-data\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034546    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-kube-api-access-8dc2x\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-kube-api-access-8dc2x\") pod \"7264bb7f-0dac-420c-8201-37a8af95e77b\" (UID: \"7264bb7f-0dac-420c-8201-37a8af95e77b\") "
Nov 03 12:28:24.034794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034541    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-certs" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:24.034794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034782    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-data" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:24.034941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034562    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7264bb7f-0dac-420c-8201-37a8af95e77b-istiod-ca-cert" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:24.034941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.034750    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-socket" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:24.035211 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.035187    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.035296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.035217    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.035296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.035233    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.035296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.035248    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7264bb7f-0dac-420c-8201-37a8af95e77b-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.035296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.035270    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.036444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.036416    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-envoy" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:24.036579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.036553    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-podinfo" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:28:24.036637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.036584    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-token" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:24.036754 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.036735    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-kube-api-access-8dc2x" pod "7264bb7f-0dac-420c-8201-37a8af95e77b" (UID: "7264bb7f-0dac-420c-8201-37a8af95e77b"). InnerVolumeSpecName "kube-api-access-8dc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:24.136514 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.136488    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.136634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.136516    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.136634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.136531    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7264bb7f-0dac-420c-8201-37a8af95e77b-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.136634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.136545    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dc2x\" (UniqueName: \"kubernetes.io/projected/7264bb7f-0dac-420c-8201-37a8af95e77b-kube-api-access-8dc2x\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:24.371209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.371121    4909 generic.go:356] "Generic (PLEG): container finished" podID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerID="f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3" exitCode=0
Nov 03 12:28:24.371209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.371180    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"
Nov 03 12:28:24.371438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.371202    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" event={"ID":"7264bb7f-0dac-420c-8201-37a8af95e77b","Type":"ContainerDied","Data":"f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3"}
Nov 03 12:28:24.371438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.371278    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq" event={"ID":"7264bb7f-0dac-420c-8201-37a8af95e77b","Type":"ContainerDied","Data":"d0315a1f637ef221e85f7e3ce0f6fb6b07ccbfc31dddc02b18edf5b5f01f7d3d"}
Nov 03 12:28:24.371438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.371307    4909 scope.go:122] "RemoveContainer" containerID="f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3"
Nov 03 12:28:24.382465 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.382446    4909 scope.go:122] "RemoveContainer" containerID="cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70"
Nov 03 12:28:24.392198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.392180    4909 scope.go:122] "RemoveContainer" containerID="f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3"
Nov 03 12:28:24.392479 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:28:24.392457    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3\": container with ID starting with f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3 not found: ID does not exist" containerID="f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3"
Nov 03 12:28:24.392560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.392492    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3"} err="failed to get container status \"f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3\": rpc error: code = NotFound desc = could not find container \"f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3\": container with ID starting with f452b9f364869316b06e28c88dab581e4c0e5870f2958c681722b278fe24d4e3 not found: ID does not exist"
Nov 03 12:28:24.392560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.392518    4909 scope.go:122] "RemoveContainer" containerID="cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70"
Nov 03 12:28:24.392791 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:28:24.392768    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70\": container with ID starting with cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70 not found: ID does not exist" containerID="cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70"
Nov 03 12:28:24.392894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.392795    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70"} err="failed to get container status \"cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70\": rpc error: code = NotFound desc = could not find container \"cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70\": container with ID starting with cd3db4e4c00e673e6fc5c013698410856af4396adfed62311a5d60dd0b227c70 not found: ID does not exist"
Nov 03 12:28:24.523467 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.523431    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"]
Nov 03 12:28:24.527949 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:24.527922    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-980a31-openshift-default-594595fdc7-fbbsq"]
Nov 03 12:28:25.123300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:25.123272    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" path="/var/lib/kubelet/pods/7264bb7f-0dac-420c-8201-37a8af95e77b/volumes"
Nov 03 12:28:39.563874 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:28:39.563838    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:28:42.919063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:42.919013    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"]
Nov 03 12:28:42.935522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:42.935237    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy" containerID="cri-o://7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5" gracePeriod=30
Nov 03 12:28:43.041660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:43.041623    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:28:48.216093 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.216069    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:28:48.239060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239011    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-token\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-token\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239056    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istiod-ca-cert\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239092    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-envoy\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239193    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-podinfo\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239261    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-credential-socket\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239301    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-kube-api-access-7rwpx\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-kube-api-access-7rwpx\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239339    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-data\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239381    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-socket\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239501    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-certs\") pod \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\" (UID: \"cd432dc4-ee39-4b4a-a314-e37e10c834a0\") "
Nov 03 12:28:48.239622 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239545    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-credential-socket" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:48.239692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239617    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istiod-ca-cert" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:48.239692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239642    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-data" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:48.239864 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239819    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-socket" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:48.239939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.239889    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-certs" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:48.240107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.240085    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.240178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.240113    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.240178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.240128    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.240178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.240142    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.240178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.240156    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.241361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.241338    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-envoy" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:48.241498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.241462    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-podinfo" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:28:48.241560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.241522    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-kube-api-access-7rwpx" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "kube-api-access-7rwpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:48.242000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.241984    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-token" pod "cd432dc4-ee39-4b4a-a314-e37e10c834a0" (UID: "cd432dc4-ee39-4b4a-a314-e37e10c834a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:48.342444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.342414    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.342444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.342439    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.342444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.342448    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd432dc4-ee39-4b4a-a314-e37e10c834a0-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.342628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.342457    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwpx\" (UniqueName: \"kubernetes.io/projected/cd432dc4-ee39-4b4a-a314-e37e10c834a0-kube-api-access-7rwpx\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:48.532008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.531923    4909 generic.go:356] "Generic (PLEG): container finished" podID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerID="7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5" exitCode=0
Nov 03 12:28:48.532008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.531979    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"
Nov 03 12:28:48.532008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.531999    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" event={"ID":"cd432dc4-ee39-4b4a-a314-e37e10c834a0","Type":"ContainerDied","Data":"7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5"}
Nov 03 12:28:48.532276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.532058    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z" event={"ID":"cd432dc4-ee39-4b4a-a314-e37e10c834a0","Type":"ContainerDied","Data":"3de84bcc90d2e951fa2a83d40d2b9ba57ccd59d1b62533270e6b50dcfbc84134"}
Nov 03 12:28:48.532276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.532086    4909 scope.go:122] "RemoveContainer" containerID="7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5"
Nov 03 12:28:48.542569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.542550    4909 scope.go:122] "RemoveContainer" containerID="7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5"
Nov 03 12:28:48.542930 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:28:48.542911    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5\": container with ID starting with 7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5 not found: ID does not exist" containerID="7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5"
Nov 03 12:28:48.542979 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.542942    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5"} err="failed to get container status \"7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5\": rpc error: code = NotFound desc = could not find container \"7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5\": container with ID starting with 7d258ed765245184d764d645e369e0c013051d895af194f467e3e248341f31b5 not found: ID does not exist"
Nov 03 12:28:48.684365 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.684330    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"]
Nov 03 12:28:48.688623 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:48.688597    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-kjx4z"]
Nov 03 12:28:49.124121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:28:49.124092    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" path="/var/lib/kubelet/pods/cd432dc4-ee39-4b4a-a314-e37e10c834a0/volumes"
Nov 03 12:29:09.565618 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:29:09.565571    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:29:10.994040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.994004    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"]
Nov 03 12:29:10.994884 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.994859    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy"
Nov 03 12:29:10.995004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.994884    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy"
Nov 03 12:29:10.995004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.994914    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy"
Nov 03 12:29:10.995004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.994923    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy"
Nov 03 12:29:10.995004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.994941    4909 cpu_manager.go:414] "containerMap: removing container" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy"
Nov 03 12:29:10.995004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.994949    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy"
Nov 03 12:29:10.995299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.995131    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy"
Nov 03 12:29:10.995299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.995146    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7264bb7f-0dac-420c-8201-37a8af95e77b" containerName="istio-proxy"
Nov 03 12:29:10.995299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:10.995158    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd432dc4-ee39-4b4a-a314-e37e10c834a0" containerName="istio-proxy"
Nov 03 12:29:11.000592 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.000567    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.097737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.097677    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"]
Nov 03 12:29:11.134150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.134112    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.134423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.134392    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.134630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.134602    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvxz\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-kube-api-access-jjvxz\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.134811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.134785    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.134963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.134945    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.135185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.135169    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.135355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.135320    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.135448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.135400    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.135448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.135433    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.236998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.236961    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237016    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237052    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237073    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237121    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237160    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237241    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvxz\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-kube-api-access-jjvxz\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237293    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237331    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237431    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237683 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237472    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.237683 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.237555    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.238145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.238117    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.238272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.238151    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.240191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.240164    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.262479 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.262411    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.291273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.291212    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.292379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.292323    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvxz\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-kube-api-access-jjvxz\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.316101 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.316072    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:11.769125 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.769100    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"]
Nov 03 12:29:11.770533 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:29:11.770506    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fb80b6_94a2_47c4_a1e8_de75b1ad3e92.slice/crio-636024e9146b121e937a1a242611c4df3020f2e5333a430d7fc2dedda6db2f65 WatchSource:0}: Error finding container 636024e9146b121e937a1a242611c4df3020f2e5333a430d7fc2dedda6db2f65: Status 404 returned error can't find the container with id 636024e9146b121e937a1a242611c4df3020f2e5333a430d7fc2dedda6db2f65
Nov 03 12:29:11.773148 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.773116    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:11.773252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.773193    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:11.773252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:11.773240    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:12.709355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.709313    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" event={"ID":"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92","Type":"ContainerStarted","Data":"fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186"}
Nov 03 12:29:12.709899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.709376    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" event={"ID":"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92","Type":"ContainerStarted","Data":"636024e9146b121e937a1a242611c4df3020f2e5333a430d7fc2dedda6db2f65"}
Nov 03 12:29:12.710947 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.710927    4909 generic.go:356] "Generic (PLEG): container finished" podID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerID="04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058" exitCode=0
Nov 03 12:29:12.711028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.710966    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" event={"ID":"c91fdb8b-3546-4258-9068-74b6b5a7c2d8","Type":"ContainerDied","Data":"04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058"}
Nov 03 12:29:12.711299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.711287    4909 scope.go:122] "RemoveContainer" containerID="04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058"
Nov 03 12:29:12.712584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.712558    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:12.712634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.712622    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:12.712681 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.712669    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:12.905186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:12.905121    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podStartSLOduration=2.905102838 podStartE2EDuration="2.905102838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:12.862219392 +0000 UTC m=+2301.874686056" watchObservedRunningTime="2026-11-03 12:29:12.905102838 +0000 UTC m=+2301.917569542"
Nov 03 12:29:13.316415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:13.316381    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:13.317933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:13.317906    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:29:13.318067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:13.317957    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 12:29:13.718967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:13.718931    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" event={"ID":"c91fdb8b-3546-4258-9068-74b6b5a7c2d8","Type":"ContainerStarted","Data":"5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a"}
Nov 03 12:29:13.721382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:13.721357    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:29:13.721522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:13.721388    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 12:29:13.721522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:13.721441    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 12:29:14.317432 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:14.317393    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:29:14.317647 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:14.317456    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 12:29:14.483724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:14.483153    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"]
Nov 03 12:29:14.483724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:14.483409    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy" containerID="cri-o://01dc6c2b925d73b7244e00a3885c112de23f719bcc33317b794851ad2daf102f" gracePeriod=30
Nov 03 12:29:14.727223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:14.727188    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 12:29:14.727775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:14.727239    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 12:29:15.316890 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:15.316858    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:29:15.317062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:15.316915    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 12:29:15.723721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:15.723078    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 12:29:15.723721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:15.723139    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 12:29:16.317120 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:16.317088    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:29:16.317598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:16.317574    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 12:29:16.722232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:16.722190    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 12:29:16.722406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:16.722237    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 12:29:17.316937 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:17.316906    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:29:17.317176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:17.317147    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 12:29:17.721868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:17.721830    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 12:29:17.722043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:17.721888    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 12:29:18.316410 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:18.316374    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:29:18.316623 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:18.316444    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 12:29:18.722045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:18.722012    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 12:29:18.722434 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:18.722070    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 12:29:19.705819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.705782    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:19.706413 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.706334    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:19.707395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.707374    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:29:19.727645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.727619    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:29:19.728080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.727950    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:29:19.728728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.728686    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:29:19.809499 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.809423    4909 generic.go:356] "Generic (PLEG): container finished" podID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerID="01dc6c2b925d73b7244e00a3885c112de23f719bcc33317b794851ad2daf102f" exitCode=0
Nov 03 12:29:19.809646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:19.809495    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" event={"ID":"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330","Type":"ContainerDied","Data":"01dc6c2b925d73b7244e00a3885c112de23f719bcc33317b794851ad2daf102f"}
Nov 03 12:29:20.181891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.181865    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:29:20.338155 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338062    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-podinfo\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338155 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338124    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istiod-ca-cert\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338199    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-envoy\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338257    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-credential-socket\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338422    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-kube-api-access-6np7t\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-kube-api-access-6np7t\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338476    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-socket\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338521    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-certs\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338552    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-token\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-token\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338579    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-credential-socket" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:20.338631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338581    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istiod-ca-cert" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:20.338631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338585    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-data\") pod \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\" (UID: \"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330\") "
Nov 03 12:29:20.338910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338812    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-data" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:20.338910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338826    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-certs" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:20.339056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.338994    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-socket" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:20.339316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.339293    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.339417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.339329    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.339417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.339347    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.339417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.339362    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.339417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.339376    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.340762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.340735    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-podinfo" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:29:20.340907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.340862    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-envoy" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:20.340967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.340915    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-kube-api-access-6np7t" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "kube-api-access-6np7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:20.341112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.341093    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-token" pod "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" (UID: "ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:20.440174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.440143    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6np7t\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-kube-api-access-6np7t\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.440174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.440174    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.440174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.440184    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.440412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.440193    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.816588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.816550    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx" event={"ID":"ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330","Type":"ContainerDied","Data":"9b792d8b80474076544ff255acce2beab2a21b5b071d900d79668b82f9399ffc"}
Nov 03 12:29:20.816588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.816582    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"
Nov 03 12:29:20.817115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.816627    4909 scope.go:122] "RemoveContainer" containerID="01dc6c2b925d73b7244e00a3885c112de23f719bcc33317b794851ad2daf102f"
Nov 03 12:29:20.969194 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:20.969154    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"]
Nov 03 12:29:21.015802 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:21.015768    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-drrxx"]
Nov 03 12:29:21.131673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:21.131593    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" path="/var/lib/kubelet/pods/ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330/volumes"
Nov 03 12:29:29.040752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:29.040717    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:29:39.568889 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:29:39.568845    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:29:39.605491 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.605434    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"]
Nov 03 12:29:39.606068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.606046    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy"
Nov 03 12:29:39.606068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.606067    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy"
Nov 03 12:29:39.606246 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.606181    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3af52c-4fab-4a6e-b3b4-fd75ea7b7330" containerName="istio-proxy"
Nov 03 12:29:39.611100 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.611082    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.669095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.669031    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2daac7-openshift-default-dockercfg-bxxtr\""
Nov 03 12:29:39.735625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.735587    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-certs\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.736046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736018    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-podinfo\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.736203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736188    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4q6\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-kube-api-access-gw4q6\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.736317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736304    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-data\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.736454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736439    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b72109-dd79-4824-a3de-1c1b87f520f8-istiod-ca-cert\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.736600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736586    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-socket\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.736763 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736747    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-envoy\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.736859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736761    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"]
Nov 03 12:29:39.736997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.736916    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-credential-socket\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.737199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.737181    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-token\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.856036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.855946    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-data\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.856850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.856806    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b72109-dd79-4824-a3de-1c1b87f520f8-istiod-ca-cert\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.857920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.857903    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-socket\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.858096 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.858080    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-envoy\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.858215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.858202    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-credential-socket\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.858673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.858656    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-token\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.858910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.858894    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-certs\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.860264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.860245    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-podinfo\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.860432 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.860416    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4q6\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-kube-api-access-gw4q6\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.860591 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.857834    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-data\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.875156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.859971    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-certs\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.875353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.860197    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-credential-socket\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.875353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.859029    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-socket\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.875603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.857632    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b72109-dd79-4824-a3de-1c1b87f520f8-istiod-ca-cert\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.875759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.875736    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-envoy\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.883186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.883159    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-podinfo\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.911544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.911516    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-token\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:39.930085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:39.930057    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4q6\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-kube-api-access-gw4q6\") pod \"e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") " pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:40.224296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:40.224266    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:40.690026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:40.689999    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"]
Nov 03 12:29:40.694926 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:29:40.694890    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b72109_dd79_4824_a3de_1c1b87f520f8.slice/crio-412332a0c111bb3794a5a1b81c292869020a0d5e9caa97a9a33a83aa39e7bbc7 WatchSource:0}: Error finding container 412332a0c111bb3794a5a1b81c292869020a0d5e9caa97a9a33a83aa39e7bbc7: Status 404 returned error can't find the container with id 412332a0c111bb3794a5a1b81c292869020a0d5e9caa97a9a33a83aa39e7bbc7
Nov 03 12:29:40.698337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:40.698304    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:40.698408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:40.698371    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:40.698408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:40.698399    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:40.950037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:40.949952    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" event={"ID":"04b72109-dd79-4824-a3de-1c1b87f520f8","Type":"ContainerStarted","Data":"18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4"}
Nov 03 12:29:40.950037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:40.950014    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" event={"ID":"04b72109-dd79-4824-a3de-1c1b87f520f8","Type":"ContainerStarted","Data":"412332a0c111bb3794a5a1b81c292869020a0d5e9caa97a9a33a83aa39e7bbc7"}
Nov 03 12:29:41.058372 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:41.058047    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podStartSLOduration=2.058013213 podStartE2EDuration="2.058013213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:41.056406039 +0000 UTC m=+2330.068872714" watchObservedRunningTime="2026-11-03 12:29:41.058013213 +0000 UTC m=+2330.070479877"
Nov 03 12:29:41.228187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:41.228095    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:41.234591 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:41.234566    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:41.957986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:41.957955    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:41.958984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:41.958962    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:29:45.641522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:45.641486    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"]
Nov 03 12:29:54.315346 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.315309    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"]
Nov 03 12:29:54.315818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.315513    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299" podUID="39c4316d-2981-4749-b91e-e27693344729" containerName="maas-api" containerID="cri-o://f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a" gracePeriod=30
Nov 03 12:29:54.633644 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.633620    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:29:54.767506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.767464    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls\") pod \"39c4316d-2981-4749-b91e-e27693344729\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") "
Nov 03 12:29:54.767786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.767568    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c4316d-2981-4749-b91e-e27693344729-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4316d-2981-4749-b91e-e27693344729-openshift-service-ca\") pod \"39c4316d-2981-4749-b91e-e27693344729\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") "
Nov 03 12:29:54.767786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.767736    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c4316d-2981-4749-b91e-e27693344729-kube-api-access-t52mg\" (UniqueName: \"kubernetes.io/projected/39c4316d-2981-4749-b91e-e27693344729-kube-api-access-t52mg\") pod \"39c4316d-2981-4749-b91e-e27693344729\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") "
Nov 03 12:29:54.767959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.767787    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-metrics-certs\") pod \"39c4316d-2981-4749-b91e-e27693344729\" (UID: \"39c4316d-2981-4749-b91e-e27693344729\") "
Nov 03 12:29:54.768062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.768018    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c4316d-2981-4749-b91e-e27693344729-openshift-service-ca" pod "39c4316d-2981-4749-b91e-e27693344729" (UID: "39c4316d-2981-4749-b91e-e27693344729"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:54.768343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.768321    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4316d-2981-4749-b91e-e27693344729-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:54.770387 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.770355    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls" pod "39c4316d-2981-4749-b91e-e27693344729" (UID: "39c4316d-2981-4749-b91e-e27693344729"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:54.770598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.770572    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c4316d-2981-4749-b91e-e27693344729-kube-api-access-t52mg" pod "39c4316d-2981-4749-b91e-e27693344729" (UID: "39c4316d-2981-4749-b91e-e27693344729"). InnerVolumeSpecName "kube-api-access-t52mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:54.771138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.771118    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-metrics-certs" pod "39c4316d-2981-4749-b91e-e27693344729" (UID: "39c4316d-2981-4749-b91e-e27693344729"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:54.872781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.870858    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52mg\" (UniqueName: \"kubernetes.io/projected/39c4316d-2981-4749-b91e-e27693344729-kube-api-access-t52mg\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:54.872781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.870898    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:54.872781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:54.870914    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c4316d-2981-4749-b91e-e27693344729-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:55.101856 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.101815    4909 generic.go:356] "Generic (PLEG): container finished" podID="39c4316d-2981-4749-b91e-e27693344729" containerID="f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a" exitCode=0
Nov 03 12:29:55.101856 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.101857    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"
Nov 03 12:29:55.102048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.101880    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299" event={"ID":"39c4316d-2981-4749-b91e-e27693344729","Type":"ContainerDied","Data":"f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a"}
Nov 03 12:29:55.102048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.101925    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299" event={"ID":"39c4316d-2981-4749-b91e-e27693344729","Type":"ContainerDied","Data":"01fbc59925f9f0888e7266c2e79846b4512b87858d5558c5c6f17cde22be2c50"}
Nov 03 12:29:55.102048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.101950    4909 scope.go:122] "RemoveContainer" containerID="f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a"
Nov 03 12:29:55.112596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.112577    4909 scope.go:122] "RemoveContainer" containerID="f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a"
Nov 03 12:29:55.112905 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:29:55.112887    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a\": container with ID starting with f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a not found: ID does not exist" containerID="f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a"
Nov 03 12:29:55.112974 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.112913    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a"} err="failed to get container status \"f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a\": rpc error: code = NotFound desc = could not find container \"f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a\": container with ID starting with f97fc90b8e9cdeab25cfb8469cbdbf95540d3ef1ae182c19cf9080087886276a not found: ID does not exist"
Nov 03 12:29:55.325605 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.325569    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"]
Nov 03 12:29:55.328287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:55.328265    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b6db28-6d78fdfc9b-vq299"]
Nov 03 12:29:56.358961 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.358921    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"]
Nov 03 12:29:56.367723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.360518    4909 cpu_manager.go:414] "containerMap: removing container" podUID="39c4316d-2981-4749-b91e-e27693344729" containerName="maas-api"
Nov 03 12:29:56.368968 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.368941    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c4316d-2981-4749-b91e-e27693344729" containerName="maas-api"
Nov 03 12:29:56.369341 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.369327    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c4316d-2981-4749-b91e-e27693344729" containerName="maas-api"
Nov 03 12:29:56.378399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.378379    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.473497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.473463    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.473834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.473807    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40ca4ba-5907-469a-9b72-86e2cf143699-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.474030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.474012    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.474216 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.474197    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sts\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-kube-api-access-98sts\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.474385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.474368    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.474560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.474543    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.474732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.474713    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.474947 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.474930    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.475221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.475200    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.475821 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.475800    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"]
Nov 03 12:29:56.576575 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576443    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.576783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576611    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.576783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576667    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.576783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576733    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40ca4ba-5907-469a-9b72-86e2cf143699-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.576783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576762    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.577047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576802    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sts\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-kube-api-access-98sts\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.577047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576837    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.577047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576883    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.577047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.576913    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.577515 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.577473    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.578132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.577803    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.578666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.578632    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.579367 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.579339    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40ca4ba-5907-469a-9b72-86e2cf143699-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.580048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.580020    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.585498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.584231    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.600588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.586239    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.630443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.630374    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.633778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.633753    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sts\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-kube-api-access-98sts\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:56.735118 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:56.735086    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:57.123822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:57.123791    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c4316d-2981-4749-b91e-e27693344729" path="/var/lib/kubelet/pods/39c4316d-2981-4749-b91e-e27693344729/volumes"
Nov 03 12:29:57.194017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:57.193982    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"]
Nov 03 12:29:57.229104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:57.225843    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:57.229104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:57.225943    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:57.229104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:57.225989    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:29:58.123536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:58.123502    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" event={"ID":"f40ca4ba-5907-469a-9b72-86e2cf143699","Type":"ContainerStarted","Data":"927effa03de3dd5ae6bb4e1558a2055853bc2fc92350f63b7a225876bd83a580"}
Nov 03 12:29:58.123931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:58.123555    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" event={"ID":"f40ca4ba-5907-469a-9b72-86e2cf143699","Type":"ContainerStarted","Data":"b8e70d8bf55a19ea471a35b38dc0b54e57879bd3b7f8edcc2cf44a61118724d2"}
Nov 03 12:29:58.278311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:58.278258    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podStartSLOduration=2.2782442769999998 podStartE2EDuration="2.278244277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:58.230036309 +0000 UTC m=+2347.242502972" watchObservedRunningTime="2026-11-03 12:29:58.278244277 +0000 UTC m=+2347.290710997"
Nov 03 12:29:58.735646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:58.735620    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:29:58.737275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:58.737251    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 12:29:58.737387 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:58.737297    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 12:29:59.735547 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:59.735510    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 12:29:59.735956 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:29:59.735568    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 12:30:00.176399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.176306    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"]
Nov 03 12:30:00.180573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.180548    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"]
Nov 03 12:30:00.180736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.180657    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.184387 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.184366    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.197226 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.197205    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"]
Nov 03 12:30:00.201851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.201829    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.228837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.228810    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:30:00.308737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.303163    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"]
Nov 03 12:30:00.308737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.307643    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"]
Nov 03 12:30:00.334520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.334490    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59812dc6-823b-4b54-acef-1b7c1fccb72c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.334673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.334527    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qcv\" (UniqueName: \"kubernetes.io/projected/8b76a780-e28f-4123-8cff-5435c2d959f3-kube-api-access-g2qcv\") pod \"maas-api-key-cleanup-29895150-tq2gf\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.334673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.334571    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b76a780-e28f-4123-8cff-5435c2d959f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895150-tq2gf\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.334805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.334689    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xxbx\" (UniqueName: \"kubernetes.io/projected/59812dc6-823b-4b54-acef-1b7c1fccb72c-kube-api-access-4xxbx\") pod \"maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.334854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.334838    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19130dfa-2a11-4bde-b1f4-97536999d294-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.334924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.334910    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2f7v\" (UniqueName: \"kubernetes.io/projected/19130dfa-2a11-4bde-b1f4-97536999d294-kube-api-access-r2f7v\") pod \"maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.374410 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.374381    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"]
Nov 03 12:30:00.436139 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436056    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xxbx\" (UniqueName: \"kubernetes.io/projected/59812dc6-823b-4b54-acef-1b7c1fccb72c-kube-api-access-4xxbx\") pod \"maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.436281 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436178    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19130dfa-2a11-4bde-b1f4-97536999d294-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.436321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436279    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2f7v\" (UniqueName: \"kubernetes.io/projected/19130dfa-2a11-4bde-b1f4-97536999d294-kube-api-access-r2f7v\") pod \"maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.436365 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436316    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59812dc6-823b-4b54-acef-1b7c1fccb72c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.436365 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436342    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qcv\" (UniqueName: \"kubernetes.io/projected/8b76a780-e28f-4123-8cff-5435c2d959f3-kube-api-access-g2qcv\") pod \"maas-api-key-cleanup-29895150-tq2gf\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.436476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436446    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b76a780-e28f-4123-8cff-5435c2d959f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895150-tq2gf\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.436909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436884    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19130dfa-2a11-4bde-b1f4-97536999d294-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.437029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.436997    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59812dc6-823b-4b54-acef-1b7c1fccb72c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.437080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.437037    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b76a780-e28f-4123-8cff-5435c2d959f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895150-tq2gf\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.491452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.491422    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xxbx\" (UniqueName: \"kubernetes.io/projected/59812dc6-823b-4b54-acef-1b7c1fccb72c-kube-api-access-4xxbx\") pod \"maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.491647 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.491624    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qcv\" (UniqueName: \"kubernetes.io/projected/8b76a780-e28f-4123-8cff-5435c2d959f3-kube-api-access-g2qcv\") pod \"maas-api-key-cleanup-29895150-tq2gf\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.492090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.492073    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2f7v\" (UniqueName: \"kubernetes.io/projected/19130dfa-2a11-4bde-b1f4-97536999d294-kube-api-access-r2f7v\") pod \"maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.498953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.498933    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:00.505202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.505183    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:00.516090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.516069    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:00.671064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.670879    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj" podUID="83aab95e-fca5-4637-a151-85bd7566b3c5" containerName="main" containerID="cri-o://3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520" gracePeriod=45
Nov 03 12:30:00.735826 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.735769    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 12:30:00.736345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.735831    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 12:30:00.835733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.835670    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj" podUID="83aab95e-fca5-4637-a151-85bd7566b3c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": dial tcp 10.132.0.112:8000: connect: connection refused"
Nov 03 12:30:00.904021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.903976    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"]
Nov 03 12:30:00.910425 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:30:00.910397    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19130dfa_2a11_4bde_b1f4_97536999d294.slice/crio-1f625362ba68077c27e06975a73ee774b0fa6bbc07467880ab2eafe99e444750 WatchSource:0}: Error finding container 1f625362ba68077c27e06975a73ee774b0fa6bbc07467880ab2eafe99e444750: Status 404 returned error can't find the container with id 1f625362ba68077c27e06975a73ee774b0fa6bbc07467880ab2eafe99e444750
Nov 03 12:30:00.967444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.967124    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"]
Nov 03 12:30:00.971398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.971373    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"]
Nov 03 12:30:00.972402 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:30:00.972376    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76a780_e28f_4123_8cff_5435c2d959f3.slice/crio-f99d60ec70f7281213790ca3729f5256abfaa6fae42b6c4cfd7b0024d3018fca WatchSource:0}: Error finding container f99d60ec70f7281213790ca3729f5256abfaa6fae42b6c4cfd7b0024d3018fca: Status 404 returned error can't find the container with id f99d60ec70f7281213790ca3729f5256abfaa6fae42b6c4cfd7b0024d3018fca
Nov 03 12:30:00.978068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:00.978045    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:30:01.169457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.169423    4909 generic.go:356] "Generic (PLEG): container finished" podID="83aab95e-fca5-4637-a151-85bd7566b3c5" containerID="3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520" exitCode=0
Nov 03 12:30:01.169629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.169590    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"
Nov 03 12:30:01.170505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.170114    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj" event={"ID":"83aab95e-fca5-4637-a151-85bd7566b3c5","Type":"ContainerDied","Data":"3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520"}
Nov 03 12:30:01.170505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.170166    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj" event={"ID":"83aab95e-fca5-4637-a151-85bd7566b3c5","Type":"ContainerDied","Data":"2b7372755edcceefe55a49eab840a354edc0d42ed8c308421d166c9091328987"}
Nov 03 12:30:01.170505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.170190    4909 scope.go:122] "RemoveContainer" containerID="3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520"
Nov 03 12:30:01.172975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.172944    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" event={"ID":"19130dfa-2a11-4bde-b1f4-97536999d294","Type":"ContainerStarted","Data":"1f625362ba68077c27e06975a73ee774b0fa6bbc07467880ab2eafe99e444750"}
Nov 03 12:30:01.174148 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.174125    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83aab95e-fca5-4637-a151-85bd7566b3c5-kube-api-access-29p9j\" (UniqueName: \"kubernetes.io/projected/83aab95e-fca5-4637-a151-85bd7566b3c5-kube-api-access-29p9j\") pod \"83aab95e-fca5-4637-a151-85bd7566b3c5\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") "
Nov 03 12:30:01.174333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.174309    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-model-cache\") pod \"83aab95e-fca5-4637-a151-85bd7566b3c5\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") "
Nov 03 12:30:01.174442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.174370    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83aab95e-fca5-4637-a151-85bd7566b3c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/83aab95e-fca5-4637-a151-85bd7566b3c5-tls-certs\") pod \"83aab95e-fca5-4637-a151-85bd7566b3c5\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") "
Nov 03 12:30:01.175356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.175285    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-model-cache" pod "83aab95e-fca5-4637-a151-85bd7566b3c5" (UID: "83aab95e-fca5-4637-a151-85bd7566b3c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:01.175476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.175456    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-tmp-dir\") pod \"83aab95e-fca5-4637-a151-85bd7566b3c5\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") "
Nov 03 12:30:01.176289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.175503    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-tmp-dir" pod "83aab95e-fca5-4637-a151-85bd7566b3c5" (UID: "83aab95e-fca5-4637-a151-85bd7566b3c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:01.176289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.176082    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-dshm\") pod \"83aab95e-fca5-4637-a151-85bd7566b3c5\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") "
Nov 03 12:30:01.176289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.176151    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-home\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-home\") pod \"83aab95e-fca5-4637-a151-85bd7566b3c5\" (UID: \"83aab95e-fca5-4637-a151-85bd7566b3c5\") "
Nov 03 12:30:01.176289 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.175721    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" event={"ID":"8b76a780-e28f-4123-8cff-5435c2d959f3","Type":"ContainerStarted","Data":"f99d60ec70f7281213790ca3729f5256abfaa6fae42b6c4cfd7b0024d3018fca"}
Nov 03 12:30:01.177103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.176532    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-home" pod "83aab95e-fca5-4637-a151-85bd7566b3c5" (UID: "83aab95e-fca5-4637-a151-85bd7566b3c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:01.177103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.176756    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:01.177103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.176778    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:01.177103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.176792    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:01.178651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.178600    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" event={"ID":"59812dc6-823b-4b54-acef-1b7c1fccb72c","Type":"ContainerStarted","Data":"641a7b5bf7bc4e6fdf78675bb73e6b30143ba91e2d08b3db3192b695f49263a9"}
Nov 03 12:30:01.178772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.178738    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-dshm" pod "83aab95e-fca5-4637-a151-85bd7566b3c5" (UID: "83aab95e-fca5-4637-a151-85bd7566b3c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:01.178917 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.178571    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83aab95e-fca5-4637-a151-85bd7566b3c5-kube-api-access-29p9j" pod "83aab95e-fca5-4637-a151-85bd7566b3c5" (UID: "83aab95e-fca5-4637-a151-85bd7566b3c5"). InnerVolumeSpecName "kube-api-access-29p9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:01.179268 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.179247    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83aab95e-fca5-4637-a151-85bd7566b3c5-tls-certs" pod "83aab95e-fca5-4637-a151-85bd7566b3c5" (UID: "83aab95e-fca5-4637-a151-85bd7566b3c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:01.185816 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.185799    4909 scope.go:122] "RemoveContainer" containerID="3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520"
Nov 03 12:30:01.186119 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:01.186098    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520\": container with ID starting with 3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520 not found: ID does not exist" containerID="3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520"
Nov 03 12:30:01.186197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.186142    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520"} err="failed to get container status \"3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520\": rpc error: code = NotFound desc = could not find container \"3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520\": container with ID starting with 3b0c100d371f32326777b9583855af1f95122ec1aef3e4489fc87284979c3520 not found: ID does not exist"
Nov 03 12:30:01.280854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.280821    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83aab95e-fca5-4637-a151-85bd7566b3c5-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:01.280989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.280857    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29p9j\" (UniqueName: \"kubernetes.io/projected/83aab95e-fca5-4637-a151-85bd7566b3c5-kube-api-access-29p9j\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:01.280989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.280874    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83aab95e-fca5-4637-a151-85bd7566b3c5-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:01.581548 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.581463    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"]
Nov 03 12:30:01.584170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.584142    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2f3eb2/ipp-route-2f3eb2-kserve-5574c4b7b5-kzfqj"]
Nov 03 12:30:01.736277 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.736236    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 12:30:01.736759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:01.736289    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 12:30:02.190528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.190495    4909 generic.go:356] "Generic (PLEG): container finished" podID="19130dfa-2a11-4bde-b1f4-97536999d294" containerID="f0a68c67ed654e97645d1ce01eec57a21ef065b187addb3c8384c19dea82f5f6" exitCode=60
Nov 03 12:30:02.190726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.190588    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" event={"ID":"19130dfa-2a11-4bde-b1f4-97536999d294","Type":"ContainerDied","Data":"f0a68c67ed654e97645d1ce01eec57a21ef065b187addb3c8384c19dea82f5f6"}
Nov 03 12:30:02.191014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.190991    4909 scope.go:122] "RemoveContainer" containerID="f0a68c67ed654e97645d1ce01eec57a21ef065b187addb3c8384c19dea82f5f6"
Nov 03 12:30:02.192572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.192548    4909 generic.go:356] "Generic (PLEG): container finished" podID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerID="a58179532abfcad96fca4ad81ee89b14ddefd4c798e3c83b43dc5ec80933b5b9" exitCode=60
Nov 03 12:30:02.192668 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.192628    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" event={"ID":"8b76a780-e28f-4123-8cff-5435c2d959f3","Type":"ContainerDied","Data":"a58179532abfcad96fca4ad81ee89b14ddefd4c798e3c83b43dc5ec80933b5b9"}
Nov 03 12:30:02.192871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.192850    4909 scope.go:122] "RemoveContainer" containerID="a58179532abfcad96fca4ad81ee89b14ddefd4c798e3c83b43dc5ec80933b5b9"
Nov 03 12:30:02.194498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.194446    4909 generic.go:356] "Generic (PLEG): container finished" podID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerID="fbdd7df79f948e9f0f0560e5632a9d58a5f30dcd9f656cdba6c241e5717e76cc" exitCode=60
Nov 03 12:30:02.194498 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.194473    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" event={"ID":"59812dc6-823b-4b54-acef-1b7c1fccb72c","Type":"ContainerDied","Data":"fbdd7df79f948e9f0f0560e5632a9d58a5f30dcd9f656cdba6c241e5717e76cc"}
Nov 03 12:30:02.194792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.194774    4909 scope.go:122] "RemoveContainer" containerID="fbdd7df79f948e9f0f0560e5632a9d58a5f30dcd9f656cdba6c241e5717e76cc"
Nov 03 12:30:02.736108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.736018    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 12:30:02.736108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:02.736072    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 12:30:03.125431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.125354    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83aab95e-fca5-4637-a151-85bd7566b3c5" path="/var/lib/kubelet/pods/83aab95e-fca5-4637-a151-85bd7566b3c5/volumes"
Nov 03 12:30:03.204268 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.204240    4909 generic.go:356] "Generic (PLEG): container finished" podID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerID="7a2f34742b1de1181f58f98ec2a2aa9ebe393a58b338006d76a10879e7699150" exitCode=60
Nov 03 12:30:03.204437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.204322    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" event={"ID":"8b76a780-e28f-4123-8cff-5435c2d959f3","Type":"ContainerDied","Data":"7a2f34742b1de1181f58f98ec2a2aa9ebe393a58b338006d76a10879e7699150"}
Nov 03 12:30:03.204437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.204391    4909 scope.go:122] "RemoveContainer" containerID="a58179532abfcad96fca4ad81ee89b14ddefd4c798e3c83b43dc5ec80933b5b9"
Nov 03 12:30:03.204783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.204762    4909 scope.go:122] "RemoveContainer" containerID="7a2f34742b1de1181f58f98ec2a2aa9ebe393a58b338006d76a10879e7699150"
Nov 03 12:30:03.205124 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:03.205088    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895150-tq2gf_odh-ai-gateway-infra(8b76a780-e28f-4123-8cff-5435c2d959f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3"
Nov 03 12:30:03.206271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.206251    4909 generic.go:356] "Generic (PLEG): container finished" podID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerID="964134da6be7b8ecc45d919632c53d92b59c226fa7b8a061b555716329577c93" exitCode=60
Nov 03 12:30:03.206378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.206333    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" event={"ID":"59812dc6-823b-4b54-acef-1b7c1fccb72c","Type":"ContainerDied","Data":"964134da6be7b8ecc45d919632c53d92b59c226fa7b8a061b555716329577c93"}
Nov 03 12:30:03.206587 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.206570    4909 scope.go:122] "RemoveContainer" containerID="964134da6be7b8ecc45d919632c53d92b59c226fa7b8a061b555716329577c93"
Nov 03 12:30:03.206888 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:03.206862    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb_odh-ai-gateway-infra(59812dc6-823b-4b54-acef-1b7c1fccb72c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c"
Nov 03 12:30:03.208939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.208913    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" event={"ID":"19130dfa-2a11-4bde-b1f4-97536999d294","Type":"ContainerStarted","Data":"441b333f0c81478906a0949467c188c2d84623b9d7bedb50b5b4c880e6436629"}
Nov 03 12:30:03.342751 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.342725    4909 scope.go:122] "RemoveContainer" containerID="fbdd7df79f948e9f0f0560e5632a9d58a5f30dcd9f656cdba6c241e5717e76cc"
Nov 03 12:30:03.523605 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.523539    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" podStartSLOduration=3.523519117 podStartE2EDuration="3.523519117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:03.51954289 +0000 UTC m=+2352.532009553" watchObservedRunningTime="2026-11-03 12:30:03.523519117 +0000 UTC m=+2352.535985781"
Nov 03 12:30:03.738379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.738339    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 12:30:03.738554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:03.738405    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 12:30:04.217988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.217958    4909 generic.go:356] "Generic (PLEG): container finished" podID="19130dfa-2a11-4bde-b1f4-97536999d294" containerID="441b333f0c81478906a0949467c188c2d84623b9d7bedb50b5b4c880e6436629" exitCode=60
Nov 03 12:30:04.218488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.218040    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" event={"ID":"19130dfa-2a11-4bde-b1f4-97536999d294","Type":"ContainerDied","Data":"441b333f0c81478906a0949467c188c2d84623b9d7bedb50b5b4c880e6436629"}
Nov 03 12:30:04.218488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.218103    4909 scope.go:122] "RemoveContainer" containerID="f0a68c67ed654e97645d1ce01eec57a21ef065b187addb3c8384c19dea82f5f6"
Nov 03 12:30:04.218488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.218385    4909 scope.go:122] "RemoveContainer" containerID="441b333f0c81478906a0949467c188c2d84623b9d7bedb50b5b4c880e6436629"
Nov 03 12:30:04.218771 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:04.218746    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n_odh-ai-gateway-infra(19130dfa-2a11-4bde-b1f4-97536999d294)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" podUID="19130dfa-2a11-4bde-b1f4-97536999d294"
Nov 03 12:30:04.220209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.220191    4909 scope.go:122] "RemoveContainer" containerID="7a2f34742b1de1181f58f98ec2a2aa9ebe393a58b338006d76a10879e7699150"
Nov 03 12:30:04.220489 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:04.220469    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895150-tq2gf_odh-ai-gateway-infra(8b76a780-e28f-4123-8cff-5435c2d959f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3"
Nov 03 12:30:04.221736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.221720    4909 scope.go:122] "RemoveContainer" containerID="964134da6be7b8ecc45d919632c53d92b59c226fa7b8a061b555716329577c93"
Nov 03 12:30:04.222011 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:04.221989    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb_odh-ai-gateway-infra(59812dc6-823b-4b54-acef-1b7c1fccb72c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c"
Nov 03 12:30:04.740152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.740123    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:30:04.740406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.740387    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:30:04.741055 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:04.741038    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:30:05.229637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:05.229602    4909 scope.go:122] "RemoveContainer" containerID="441b333f0c81478906a0949467c188c2d84623b9d7bedb50b5b4c880e6436629"
Nov 03 12:30:05.230026 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:05.229985    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n_odh-ai-gateway-infra(19130dfa-2a11-4bde-b1f4-97536999d294)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" podUID="19130dfa-2a11-4bde-b1f4-97536999d294"
Nov 03 12:30:09.565168 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:09.565130    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:30:11.408187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:11.408154    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"]
Nov 03 12:30:11.505186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:11.505150    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"]
Nov 03 12:30:14.137728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:14.132711    4909 scope.go:122] "RemoveContainer" containerID="964134da6be7b8ecc45d919632c53d92b59c226fa7b8a061b555716329577c93"
Nov 03 12:30:15.336601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:15.336527    4909 scope.go:122] "RemoveContainer" containerID="7a2f34742b1de1181f58f98ec2a2aa9ebe393a58b338006d76a10879e7699150"
Nov 03 12:30:15.356907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:15.356882    4909 generic.go:356] "Generic (PLEG): container finished" podID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerID="5514f4d17dd970a002a399638f4de73b46138bc7e57cfc7f67f5989048bb391e" exitCode=60
Nov 03 12:30:15.357014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:15.356957    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" event={"ID":"59812dc6-823b-4b54-acef-1b7c1fccb72c","Type":"ContainerDied","Data":"5514f4d17dd970a002a399638f4de73b46138bc7e57cfc7f67f5989048bb391e"}
Nov 03 12:30:15.357076 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:15.357034    4909 scope.go:122] "RemoveContainer" containerID="964134da6be7b8ecc45d919632c53d92b59c226fa7b8a061b555716329577c93"
Nov 03 12:30:15.357328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:15.357312    4909 scope.go:122] "RemoveContainer" containerID="5514f4d17dd970a002a399638f4de73b46138bc7e57cfc7f67f5989048bb391e"
Nov 03 12:30:15.357572 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:15.357552    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb_odh-ai-gateway-infra(59812dc6-823b-4b54-acef-1b7c1fccb72c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c"
Nov 03 12:30:16.125953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.125917    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:30:16.365176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.365147    4909 generic.go:356] "Generic (PLEG): container finished" podID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerID="edd59443a01e08c5eb664d9ee25ed8a304fc330832e42cf288c3e61ece05dd66" exitCode=60
Nov 03 12:30:16.365649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.365224    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" event={"ID":"8b76a780-e28f-4123-8cff-5435c2d959f3","Type":"ContainerDied","Data":"edd59443a01e08c5eb664d9ee25ed8a304fc330832e42cf288c3e61ece05dd66"}
Nov 03 12:30:16.365649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.365284    4909 scope.go:122] "RemoveContainer" containerID="7a2f34742b1de1181f58f98ec2a2aa9ebe393a58b338006d76a10879e7699150"
Nov 03 12:30:16.365813 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.365743    4909 scope.go:122] "RemoveContainer" containerID="edd59443a01e08c5eb664d9ee25ed8a304fc330832e42cf288c3e61ece05dd66"
Nov 03 12:30:16.366104 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:16.366075    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895150-tq2gf_odh-ai-gateway-infra(8b76a780-e28f-4123-8cff-5435c2d959f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3"
Nov 03 12:30:16.408762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.408735    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f" podUID="763e0649-90d2-4a95-9e58-a48c24514abf" containerName="payload-processing" containerID="cri-o://5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc" gracePeriod=25
Nov 03 12:30:16.471973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.471950    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"]
Nov 03 12:30:16.506227 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.506189    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z" podUID="ae043112-933c-4a53-bea8-292b0d08d034" containerName="payload-pre-processing" containerID="cri-o://00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997" gracePeriod=25
Nov 03 12:30:16.659437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.659117    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:16.702819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.702731    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:30:16.759435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.759406    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59812dc6-823b-4b54-acef-1b7c1fccb72c-kube-api-access-4xxbx\" (UniqueName: \"kubernetes.io/projected/59812dc6-823b-4b54-acef-1b7c1fccb72c-kube-api-access-4xxbx\") pod \"59812dc6-823b-4b54-acef-1b7c1fccb72c\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") "
Nov 03 12:30:16.759603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.759468    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59812dc6-823b-4b54-acef-1b7c1fccb72c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59812dc6-823b-4b54-acef-1b7c1fccb72c-maas-api-service-ca\") pod \"59812dc6-823b-4b54-acef-1b7c1fccb72c\" (UID: \"59812dc6-823b-4b54-acef-1b7c1fccb72c\") "
Nov 03 12:30:16.760485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.760449    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59812dc6-823b-4b54-acef-1b7c1fccb72c-maas-api-service-ca" pod "59812dc6-823b-4b54-acef-1b7c1fccb72c" (UID: "59812dc6-823b-4b54-acef-1b7c1fccb72c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:16.762235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.762212    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59812dc6-823b-4b54-acef-1b7c1fccb72c-kube-api-access-4xxbx" pod "59812dc6-823b-4b54-acef-1b7c1fccb72c" (UID: "59812dc6-823b-4b54-acef-1b7c1fccb72c"). InnerVolumeSpecName "kube-api-access-4xxbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:16.775634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.775614    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:30:16.860973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.860945    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae043112-933c-4a53-bea8-292b0d08d034-kube-api-access-kjcth\" (UniqueName: \"kubernetes.io/projected/ae043112-933c-4a53-bea8-292b0d08d034-kube-api-access-kjcth\") pod \"ae043112-933c-4a53-bea8-292b0d08d034\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") "
Nov 03 12:30:16.861123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861001    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae043112-933c-4a53-bea8-292b0d08d034-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae043112-933c-4a53-bea8-292b0d08d034-plugins-config-volume\") pod \"ae043112-933c-4a53-bea8-292b0d08d034\" (UID: \"ae043112-933c-4a53-bea8-292b0d08d034\") "
Nov 03 12:30:16.861123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861066    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763e0649-90d2-4a95-9e58-a48c24514abf-kube-api-access-g9gwl\" (UniqueName: \"kubernetes.io/projected/763e0649-90d2-4a95-9e58-a48c24514abf-kube-api-access-g9gwl\") pod \"763e0649-90d2-4a95-9e58-a48c24514abf\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") "
Nov 03 12:30:16.861123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861123    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763e0649-90d2-4a95-9e58-a48c24514abf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e0649-90d2-4a95-9e58-a48c24514abf-plugins-config-volume\") pod \"763e0649-90d2-4a95-9e58-a48c24514abf\" (UID: \"763e0649-90d2-4a95-9e58-a48c24514abf\") "
Nov 03 12:30:16.861442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861417    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae043112-933c-4a53-bea8-292b0d08d034-plugins-config-volume" pod "ae043112-933c-4a53-bea8-292b0d08d034" (UID: "ae043112-933c-4a53-bea8-292b0d08d034"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:16.861538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861446    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763e0649-90d2-4a95-9e58-a48c24514abf-plugins-config-volume" pod "763e0649-90d2-4a95-9e58-a48c24514abf" (UID: "763e0649-90d2-4a95-9e58-a48c24514abf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:16.861603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861561    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e0649-90d2-4a95-9e58-a48c24514abf-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:16.861603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861574    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xxbx\" (UniqueName: \"kubernetes.io/projected/59812dc6-823b-4b54-acef-1b7c1fccb72c-kube-api-access-4xxbx\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:16.861603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861583    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59812dc6-823b-4b54-acef-1b7c1fccb72c-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:16.861603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.861593    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae043112-933c-4a53-bea8-292b0d08d034-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:16.863165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.863137    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae043112-933c-4a53-bea8-292b0d08d034-kube-api-access-kjcth" pod "ae043112-933c-4a53-bea8-292b0d08d034" (UID: "ae043112-933c-4a53-bea8-292b0d08d034"). InnerVolumeSpecName "kube-api-access-kjcth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:16.863165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.863156    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763e0649-90d2-4a95-9e58-a48c24514abf-kube-api-access-g9gwl" pod "763e0649-90d2-4a95-9e58-a48c24514abf" (UID: "763e0649-90d2-4a95-9e58-a48c24514abf"). InnerVolumeSpecName "kube-api-access-g9gwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:16.969280 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.969246    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9gwl\" (UniqueName: \"kubernetes.io/projected/763e0649-90d2-4a95-9e58-a48c24514abf-kube-api-access-g9gwl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:16.969280 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:16.969280    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjcth\" (UniqueName: \"kubernetes.io/projected/ae043112-933c-4a53-bea8-292b0d08d034-kube-api-access-kjcth\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:17.120904 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.120872    4909 scope.go:122] "RemoveContainer" containerID="441b333f0c81478906a0949467c188c2d84623b9d7bedb50b5b4c880e6436629"
Nov 03 12:30:17.381286 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.381193    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"
Nov 03 12:30:17.381286 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.381221    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb" event={"ID":"59812dc6-823b-4b54-acef-1b7c1fccb72c","Type":"ContainerDied","Data":"641a7b5bf7bc4e6fdf78675bb73e6b30143ba91e2d08b3db3192b695f49263a9"}
Nov 03 12:30:17.381888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.381291    4909 scope.go:122] "RemoveContainer" containerID="5514f4d17dd970a002a399638f4de73b46138bc7e57cfc7f67f5989048bb391e"
Nov 03 12:30:17.383165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.382606    4909 generic.go:356] "Generic (PLEG): container finished" podID="763e0649-90d2-4a95-9e58-a48c24514abf" containerID="5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc" exitCode=0
Nov 03 12:30:17.383165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.382675    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"
Nov 03 12:30:17.383165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.382689    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f" event={"ID":"763e0649-90d2-4a95-9e58-a48c24514abf","Type":"ContainerDied","Data":"5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc"}
Nov 03 12:30:17.383165 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.382764    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f" event={"ID":"763e0649-90d2-4a95-9e58-a48c24514abf","Type":"ContainerDied","Data":"9fa81257418ab7a443f3c760f9e61116993d7a3129287aec29da518a69fc9611"}
Nov 03 12:30:17.385709 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.385674    4909 generic.go:356] "Generic (PLEG): container finished" podID="ae043112-933c-4a53-bea8-292b0d08d034" containerID="00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997" exitCode=0
Nov 03 12:30:17.385841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.385791    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z" event={"ID":"ae043112-933c-4a53-bea8-292b0d08d034","Type":"ContainerDied","Data":"00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997"}
Nov 03 12:30:17.385841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.385831    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z" event={"ID":"ae043112-933c-4a53-bea8-292b0d08d034","Type":"ContainerDied","Data":"dad80936b6b6cdd6e81beba1c78b63d5c9d4a5bf536ec5836af687e70d372dc7"}
Nov 03 12:30:17.385972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.385880    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"
Nov 03 12:30:17.395434 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.395413    4909 scope.go:122] "RemoveContainer" containerID="5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc"
Nov 03 12:30:17.406244 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.406225    4909 scope.go:122] "RemoveContainer" containerID="5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc"
Nov 03 12:30:17.406580 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:17.406555    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc\": container with ID starting with 5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc not found: ID does not exist" containerID="5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc"
Nov 03 12:30:17.406739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.406715    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc"} err="failed to get container status \"5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc\": rpc error: code = NotFound desc = could not find container \"5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc\": container with ID starting with 5621fdcb2e1da0fa1cfa4ba43f0bbe12c016d3581c20a40c9a05001ea11a7bcc not found: ID does not exist"
Nov 03 12:30:17.406848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.406837    4909 scope.go:122] "RemoveContainer" containerID="00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997"
Nov 03 12:30:17.418234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.418200    4909 scope.go:122] "RemoveContainer" containerID="00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997"
Nov 03 12:30:17.418539 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:17.418518    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997\": container with ID starting with 00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997 not found: ID does not exist" containerID="00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997"
Nov 03 12:30:17.418676 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.418545    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997"} err="failed to get container status \"00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997\": rpc error: code = NotFound desc = could not find container \"00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997\": container with ID starting with 00f98b685e059008b521c9dd0eeecedd9346c99775fbece06c575b4e47f80997 not found: ID does not exist"
Nov 03 12:30:17.487224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.487142    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"]
Nov 03 12:30:17.544282 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.542758    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"]
Nov 03 12:30:17.547024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.546938    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-40ffcd-69dc94f4b7-mhn8z"]
Nov 03 12:30:17.648157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.648126    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"]
Nov 03 12:30:17.655050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.654896    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-40ffcd-29895150-9ntvb"]
Nov 03 12:30:17.820220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.819101    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"]
Nov 03 12:30:17.820220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.819154    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-40ffcd-5484886ccf-c6c7f"]
Nov 03 12:30:17.860740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:17.858603    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:18.018512 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.018475    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b76a780-e28f-4123-8cff-5435c2d959f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b76a780-e28f-4123-8cff-5435c2d959f3-maas-api-service-ca\") pod \"8b76a780-e28f-4123-8cff-5435c2d959f3\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") "
Nov 03 12:30:18.020389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.020365    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b76a780-e28f-4123-8cff-5435c2d959f3-kube-api-access-g2qcv\" (UniqueName: \"kubernetes.io/projected/8b76a780-e28f-4123-8cff-5435c2d959f3-kube-api-access-g2qcv\") pod \"8b76a780-e28f-4123-8cff-5435c2d959f3\" (UID: \"8b76a780-e28f-4123-8cff-5435c2d959f3\") "
Nov 03 12:30:18.021442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.019122    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76a780-e28f-4123-8cff-5435c2d959f3-maas-api-service-ca" pod "8b76a780-e28f-4123-8cff-5435c2d959f3" (UID: "8b76a780-e28f-4123-8cff-5435c2d959f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:18.024425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.024400    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b76a780-e28f-4123-8cff-5435c2d959f3-kube-api-access-g2qcv" pod "8b76a780-e28f-4123-8cff-5435c2d959f3" (UID: "8b76a780-e28f-4123-8cff-5435c2d959f3"). InnerVolumeSpecName "kube-api-access-g2qcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:18.123077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.123029    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b76a780-e28f-4123-8cff-5435c2d959f3-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.123420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.123398    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qcv\" (UniqueName: \"kubernetes.io/projected/8b76a780-e28f-4123-8cff-5435c2d959f3-kube-api-access-g2qcv\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.416001 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.415971    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" event={"ID":"19130dfa-2a11-4bde-b1f4-97536999d294","Type":"ContainerStarted","Data":"603fc7262b54bbea933972c10d867d52e11e53af3805340b9dce7f60355c675d"}
Nov 03 12:30:18.420721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.419206    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf" event={"ID":"8b76a780-e28f-4123-8cff-5435c2d959f3","Type":"ContainerDied","Data":"f99d60ec70f7281213790ca3729f5256abfaa6fae42b6c4cfd7b0024d3018fca"}
Nov 03 12:30:18.420721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.419271    4909 scope.go:122] "RemoveContainer" containerID="edd59443a01e08c5eb664d9ee25ed8a304fc330832e42cf288c3e61ece05dd66"
Nov 03 12:30:18.420721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.419417    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"
Nov 03 12:30:18.691776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.686494    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"]
Nov 03 12:30:18.693135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:18.693111    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895150-tq2gf"]
Nov 03 12:30:19.123688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.123613    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" path="/var/lib/kubelet/pods/59812dc6-823b-4b54-acef-1b7c1fccb72c/volumes"
Nov 03 12:30:19.124075 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.124061    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763e0649-90d2-4a95-9e58-a48c24514abf" path="/var/lib/kubelet/pods/763e0649-90d2-4a95-9e58-a48c24514abf/volumes"
Nov 03 12:30:19.124438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.124424    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" path="/var/lib/kubelet/pods/8b76a780-e28f-4123-8cff-5435c2d959f3/volumes"
Nov 03 12:30:19.125118 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.125103    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae043112-933c-4a53-bea8-292b0d08d034" path="/var/lib/kubelet/pods/ae043112-933c-4a53-bea8-292b0d08d034/volumes"
Nov 03 12:30:19.438628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.438547    4909 generic.go:356] "Generic (PLEG): container finished" podID="19130dfa-2a11-4bde-b1f4-97536999d294" containerID="603fc7262b54bbea933972c10d867d52e11e53af3805340b9dce7f60355c675d" exitCode=60
Nov 03 12:30:19.439095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.438624    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" event={"ID":"19130dfa-2a11-4bde-b1f4-97536999d294","Type":"ContainerDied","Data":"603fc7262b54bbea933972c10d867d52e11e53af3805340b9dce7f60355c675d"}
Nov 03 12:30:19.439095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.438682    4909 scope.go:122] "RemoveContainer" containerID="441b333f0c81478906a0949467c188c2d84623b9d7bedb50b5b4c880e6436629"
Nov 03 12:30:19.439095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.439053    4909 scope.go:122] "RemoveContainer" containerID="603fc7262b54bbea933972c10d867d52e11e53af3805340b9dce7f60355c675d"
Nov 03 12:30:19.439407 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:19.439384    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n_odh-ai-gateway-infra(19130dfa-2a11-4bde-b1f4-97536999d294)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" podUID="19130dfa-2a11-4bde-b1f4-97536999d294"
Nov 03 12:30:19.552736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:19.552680    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"]
Nov 03 12:30:20.589797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:20.589771    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:20.656139 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:20.656111    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19130dfa-2a11-4bde-b1f4-97536999d294-kube-api-access-r2f7v\" (UniqueName: \"kubernetes.io/projected/19130dfa-2a11-4bde-b1f4-97536999d294-kube-api-access-r2f7v\") pod \"19130dfa-2a11-4bde-b1f4-97536999d294\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") "
Nov 03 12:30:20.656271 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:20.656225    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19130dfa-2a11-4bde-b1f4-97536999d294-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19130dfa-2a11-4bde-b1f4-97536999d294-maas-api-service-ca\") pod \"19130dfa-2a11-4bde-b1f4-97536999d294\" (UID: \"19130dfa-2a11-4bde-b1f4-97536999d294\") "
Nov 03 12:30:20.656595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:20.656571    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19130dfa-2a11-4bde-b1f4-97536999d294-maas-api-service-ca" pod "19130dfa-2a11-4bde-b1f4-97536999d294" (UID: "19130dfa-2a11-4bde-b1f4-97536999d294"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:20.656665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:20.656651    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19130dfa-2a11-4bde-b1f4-97536999d294-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:20.658204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:20.658178    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19130dfa-2a11-4bde-b1f4-97536999d294-kube-api-access-r2f7v" pod "19130dfa-2a11-4bde-b1f4-97536999d294" (UID: "19130dfa-2a11-4bde-b1f4-97536999d294"). InnerVolumeSpecName "kube-api-access-r2f7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:20.757686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:20.757614    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2f7v\" (UniqueName: \"kubernetes.io/projected/19130dfa-2a11-4bde-b1f4-97536999d294-kube-api-access-r2f7v\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:21.454598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:21.454572    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"
Nov 03 12:30:21.454791 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:21.454612    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n" event={"ID":"19130dfa-2a11-4bde-b1f4-97536999d294","Type":"ContainerDied","Data":"1f625362ba68077c27e06975a73ee774b0fa6bbc07467880ab2eafe99e444750"}
Nov 03 12:30:21.454791 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:21.454679    4909 scope.go:122] "RemoveContainer" containerID="603fc7262b54bbea933972c10d867d52e11e53af3805340b9dce7f60355c675d"
Nov 03 12:30:21.606947 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:21.606914    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"]
Nov 03 12:30:21.652784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:21.652756    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d25504-29895150-qg66n"]
Nov 03 12:30:23.123848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:23.123818    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" path="/var/lib/kubelet/pods/19130dfa-2a11-4bde-b1f4-97536999d294/volumes"
Nov 03 12:30:29.967968 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:29.967937    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"]
Nov 03 12:30:29.968482 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:29.968174    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" containerID="cri-o://bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d" gracePeriod=30
Nov 03 12:30:30.014493 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.014454    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"]
Nov 03 12:30:30.014778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.014754    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" containerID="cri-o://75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc" gracePeriod=30
Nov 03 12:30:30.017235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.017211    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"]
Nov 03 12:30:30.017431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.017380    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" containerID="cri-o://4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41" gracePeriod=30
Nov 03 12:30:30.060840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.060809    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"]
Nov 03 12:30:30.061239 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.061209    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" containerID="cri-o://17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130" gracePeriod=30
Nov 03 12:30:30.149733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.149599    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnmsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:30:30.149733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:30.149658    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:30:31.129439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:31.129409    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sdw94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:30:31.129836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:31.129467    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:30:32.027372 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:32.027330    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzpnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:30:32.027565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:32.027393    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:30:32.103773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:32.103743    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cgbnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:30:32.103928 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:32.103792    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:30:35.315157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.315132    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:30:35.337233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337028    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-socket\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337085    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-envoy\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337140    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-credential-socket\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337188    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-data\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337237    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e47f057-cdb7-45a5-99b9-4e16c5887078-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e47f057-cdb7-45a5-99b9-4e16c5887078-istiod-ca-cert\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337274    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-podinfo\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337299    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-token\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-token\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337326    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-kube-api-access-k8hsd\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-kube-api-access-k8hsd\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337468    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-socket" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.337690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337480    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-certs\") pod \"3e47f057-cdb7-45a5-99b9-4e16c5887078\" (UID: \"3e47f057-cdb7-45a5-99b9-4e16c5887078\") "
Nov 03 12:30:35.337874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337744    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e47f057-cdb7-45a5-99b9-4e16c5887078-istiod-ca-cert" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:35.337958 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337923    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-certs" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.338024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.337994    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-credential-socket" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.338419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.338137    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-data" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.338419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.338332    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.338419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.338352    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.338419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.338368    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.338419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.338382    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.338419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.338397    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e47f057-cdb7-45a5-99b9-4e16c5887078-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.339603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.339580    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-envoy" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.340038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.340006    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-token" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.340142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.340119    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-kube-api-access-k8hsd" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "kube-api-access-k8hsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.340228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.340212    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-podinfo" pod "3e47f057-cdb7-45a5-99b9-4e16c5887078" (UID: "3e47f057-cdb7-45a5-99b9-4e16c5887078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:30:35.426598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.426569    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:30:35.439580 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.439549    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.439738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.439582    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.439738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.439599    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8hsd\" (UniqueName: \"kubernetes.io/projected/3e47f057-cdb7-45a5-99b9-4e16c5887078-kube-api-access-k8hsd\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.439738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.439615    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e47f057-cdb7-45a5-99b9-4e16c5887078-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.441597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.441569    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:30:35.447639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.447620    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:30:35.540045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.539959    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-certs\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.540045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540011    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-socket\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.540287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540050    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-certs\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.540287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540077    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-podinfo\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.540287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540119    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istiod-ca-cert\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.540287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540164    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-kube-api-access-zpfw7\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-kube-api-access-zpfw7\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.540287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540218    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istiod-ca-cert\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.540287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540252    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-data\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.540287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540273    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-certs" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540325    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-socket\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540326    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-socket" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540358    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-certs\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540387    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-kube-api-access-bpbpj\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-kube-api-access-bpbpj\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540415    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-data\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540451    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-kube-api-access-stlv6\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-kube-api-access-stlv6\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540477    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-envoy\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540510    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-data\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540564    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-istio-token\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-istio-token\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540602    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-credential-socket\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540630    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9952d74a-882b-476d-a0cb-53d663693c1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9952d74a-882b-476d-a0cb-53d663693c1e-istiod-ca-cert\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540750    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-credential-socket\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540857    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-token\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-token\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540877    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" event={"ID":"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc","Type":"ContainerDied","Data":"4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41"}
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540892    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-credential-socket\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540928    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-envoy\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540938    4909 scope.go:122] "RemoveContainer" containerID="4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41"
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540927    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-data" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.541116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540964    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-certs" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541065    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istiod-ca-cert" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541174    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-socket" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541100    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541299    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-data" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541320    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istiod-ca-cert" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541349    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-token\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541358    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-certs" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.540854    4909 generic.go:356] "Generic (PLEG): container finished" podID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerID="4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41" exitCode=0
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541385    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-podinfo\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541437    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-envoy\") pod \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\" (UID: \"0a7f021b-e65f-4a54-ad09-69471a8d9b0a\") "
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541491    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9952d74a-882b-476d-a0cb-53d663693c1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9952d74a-882b-476d-a0cb-53d663693c1e-istio-podinfo\") pod \"9952d74a-882b-476d-a0cb-53d663693c1e\" (UID: \"9952d74a-882b-476d-a0cb-53d663693c1e\") "
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541522    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-socket\") pod \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\" (UID: \"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc\") "
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.541615    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9952d74a-882b-476d-a0cb-53d663693c1e-istiod-ca-cert" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542091    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542109    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542124    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542138    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9952d74a-882b-476d-a0cb-53d663693c1e-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.542153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542152    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542161    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-credential-socket" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542167    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542207    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542224    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542240    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542256    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542281    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv" event={"ID":"e40b512a-d8d7-4113-a10c-7b0fb47bbcbc","Type":"ContainerDied","Data":"387f89beaf8ca306e2f95271bba8d0582e15cc9ed3547342e6499a6ba4860ae9"}
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.542344    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-socket" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.543492    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-podinfo" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.543548    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-credential-socket" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.543929    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-data" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.544422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.544130    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-credential-socket" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.545143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.544792    4909 generic.go:356] "Generic (PLEG): container finished" podID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerID="bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d" exitCode=0
Nov 03 12:30:35.545143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.544864    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"
Nov 03 12:30:35.545143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.544955    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" event={"ID":"3e47f057-cdb7-45a5-99b9-4e16c5887078","Type":"ContainerDied","Data":"bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d"}
Nov 03 12:30:35.545143 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.544998    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94" event={"ID":"3e47f057-cdb7-45a5-99b9-4e16c5887078","Type":"ContainerDied","Data":"e474b9588b845dc07f7cda1a44b1eec7d5e4142809da8b66124d8fcbd395e56b"}
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548840    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9952d74a-882b-476d-a0cb-53d663693c1e-istio-podinfo" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548880    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-kube-api-access-zpfw7" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "kube-api-access-zpfw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548903    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-istio-token" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548924    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-token" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548908    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-envoy" pod "9952d74a-882b-476d-a0cb-53d663693c1e" (UID: "9952d74a-882b-476d-a0cb-53d663693c1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548913    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-kube-api-access-bpbpj" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "kube-api-access-bpbpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548926    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-token" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548954    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-kube-api-access-stlv6" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "kube-api-access-stlv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548968    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-envoy" pod "0a7f021b-e65f-4a54-ad09-69471a8d9b0a" (UID: "0a7f021b-e65f-4a54-ad09-69471a8d9b0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.548992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.548983    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-podinfo" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:30:35.549550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.549263    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-envoy" pod "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" (UID: "e40b512a-d8d7-4113-a10c-7b0fb47bbcbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.550233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.550209    4909 generic.go:356] "Generic (PLEG): container finished" podID="9952d74a-882b-476d-a0cb-53d663693c1e" containerID="75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc" exitCode=0
Nov 03 12:30:35.550361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.550247    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"
Nov 03 12:30:35.550361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.550272    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" event={"ID":"9952d74a-882b-476d-a0cb-53d663693c1e","Type":"ContainerDied","Data":"75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc"}
Nov 03 12:30:35.550361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.550324    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr" event={"ID":"9952d74a-882b-476d-a0cb-53d663693c1e","Type":"ContainerDied","Data":"ca88eb69f8294401d4fa5e6f8766f69901a23741bb90dbd3132728a64d1ef8cf"}
Nov 03 12:30:35.552007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.551985    4909 generic.go:356] "Generic (PLEG): container finished" podID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerID="17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130" exitCode=0
Nov 03 12:30:35.552112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.552022    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" event={"ID":"0a7f021b-e65f-4a54-ad09-69471a8d9b0a","Type":"ContainerDied","Data":"17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130"}
Nov 03 12:30:35.552112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.552039    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"
Nov 03 12:30:35.552112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.552061    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn" event={"ID":"0a7f021b-e65f-4a54-ad09-69471a8d9b0a","Type":"ContainerDied","Data":"53d8b3765df42951ca41f1123d8400421b037cd2b7e6d03aeb4f40019f456fb5"}
Nov 03 12:30:35.556685 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.556317    4909 scope.go:122] "RemoveContainer" containerID="4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41"
Nov 03 12:30:35.561108 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:35.561079    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41\": container with ID starting with 4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41 not found: ID does not exist" containerID="4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41"
Nov 03 12:30:35.561247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.561222    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41"} err="failed to get container status \"4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41\": rpc error: code = NotFound desc = could not find container \"4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41\": container with ID starting with 4215bc851aa5483e957981e6328a0631fd3a491239a053284f9485e586870f41 not found: ID does not exist"
Nov 03 12:30:35.561348 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.561337    4909 scope.go:122] "RemoveContainer" containerID="bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d"
Nov 03 12:30:35.574683 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.574656    4909 scope.go:122] "RemoveContainer" containerID="bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d"
Nov 03 12:30:35.575046 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:35.575025    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d\": container with ID starting with bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d not found: ID does not exist" containerID="bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d"
Nov 03 12:30:35.575140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.575056    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d"} err="failed to get container status \"bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d\": rpc error: code = NotFound desc = could not find container \"bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d\": container with ID starting with bde4e37277a1528780b53207614257ed13f6e3ff3f256606fede76312d92066d not found: ID does not exist"
Nov 03 12:30:35.575140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.575076    4909 scope.go:122] "RemoveContainer" containerID="75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc"
Nov 03 12:30:35.597880 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.597847    4909 scope.go:122] "RemoveContainer" containerID="75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc"
Nov 03 12:30:35.598285 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:35.598257    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc\": container with ID starting with 75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc not found: ID does not exist" containerID="75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc"
Nov 03 12:30:35.598384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.598298    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc"} err="failed to get container status \"75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc\": rpc error: code = NotFound desc = could not find container \"75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc\": container with ID starting with 75dfe350ddbc79a651ef6b46fdea9a60b17f207997f8149c5fa2679e2a53ddbc not found: ID does not exist"
Nov 03 12:30:35.598384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.598324    4909 scope.go:122] "RemoveContainer" containerID="17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130"
Nov 03 12:30:35.607182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.607167    4909 scope.go:122] "RemoveContainer" containerID="17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130"
Nov 03 12:30:35.607535 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:35.607506    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130\": container with ID starting with 17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130 not found: ID does not exist" containerID="17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130"
Nov 03 12:30:35.607607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.607545    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130"} err="failed to get container status \"17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130\": rpc error: code = NotFound desc = could not find container \"17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130\": container with ID starting with 17196c95cc471c6efae8e637f183bfc166d43ef4bc44d1174bac8a6f77472130 not found: ID does not exist"
Nov 03 12:30:35.643660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643636    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbpj\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-kube-api-access-bpbpj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643659    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlv6\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-kube-api-access-stlv6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643671    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643680    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643688    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643722    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643731    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643739    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643747    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9952d74a-882b-476d-a0cb-53d663693c1e-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643756    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643765    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643773    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643781    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643790    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9952d74a-882b-476d-a0cb-53d663693c1e-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643798    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643806    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f021b-e65f-4a54-ad09-69471a8d9b0a-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.643833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.643814    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfw7\" (UniqueName: \"kubernetes.io/projected/9952d74a-882b-476d-a0cb-53d663693c1e-kube-api-access-zpfw7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.893429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.893399    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"]
Nov 03 12:30:35.900773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.900742    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sdw94"]
Nov 03 12:30:35.998331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:35.998304    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"]
Nov 03 12:30:36.001064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:36.001043    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnmsr"]
Nov 03 12:30:36.149363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:36.149287    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"]
Nov 03 12:30:36.151980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:36.151955    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cgbnn"]
Nov 03 12:30:36.252149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:36.252117    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"]
Nov 03 12:30:36.255853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:36.255829    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzpnv"]
Nov 03 12:30:37.122831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:37.122802    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" path="/var/lib/kubelet/pods/0a7f021b-e65f-4a54-ad09-69471a8d9b0a/volumes"
Nov 03 12:30:37.123277 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:37.123263    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" path="/var/lib/kubelet/pods/3e47f057-cdb7-45a5-99b9-4e16c5887078/volumes"
Nov 03 12:30:37.123721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:37.123687    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" path="/var/lib/kubelet/pods/9952d74a-882b-476d-a0cb-53d663693c1e/volumes"
Nov 03 12:30:37.124430 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:37.124416    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" path="/var/lib/kubelet/pods/e40b512a-d8d7-4113-a10c-7b0fb47bbcbc/volumes"
Nov 03 12:30:39.563753 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:39.563715    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:30:41.669391 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.669360    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"]
Nov 03 12:30:41.670075 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670053    4909 cpu_manager.go:414] "containerMap: removing container" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670077    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670103    4909 cpu_manager.go:414] "containerMap: removing container" podUID="763e0649-90d2-4a95-9e58-a48c24514abf" containerName="payload-processing"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670111    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763e0649-90d2-4a95-9e58-a48c24514abf" containerName="payload-processing"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670135    4909 cpu_manager.go:414] "containerMap: removing container" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670143    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670155    4909 cpu_manager.go:414] "containerMap: removing container" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670162    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670175    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.670183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670181    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670190    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670197    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670212    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ae043112-933c-4a53-bea8-292b0d08d034" containerName="payload-pre-processing"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670221    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae043112-933c-4a53-bea8-292b0d08d034" containerName="payload-pre-processing"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670233    4909 cpu_manager.go:414] "containerMap: removing container" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670240    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670265    4909 cpu_manager.go:414] "containerMap: removing container" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670272    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670287    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670294    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670305    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670313    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670326    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670333    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670344    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670351    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670368    4909 cpu_manager.go:414] "containerMap: removing container" podUID="83aab95e-fca5-4637-a151-85bd7566b3c5" containerName="main"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670375    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83aab95e-fca5-4637-a151-85bd7566b3c5" containerName="main"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670488    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae043112-933c-4a53-bea8-292b0d08d034" containerName="payload-pre-processing"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670504    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670513    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670524    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670534    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670543    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9952d74a-882b-476d-a0cb-53d663693c1e" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670554    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e47f057-cdb7-45a5-99b9-4e16c5887078" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670562    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40b512a-d8d7-4113-a10c-7b0fb47bbcbc" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670572    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="83aab95e-fca5-4637-a151-85bd7566b3c5" containerName="main"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670580    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7f021b-e65f-4a54-ad09-69471a8d9b0a" containerName="istio-proxy"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670592    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="763e0649-90d2-4a95-9e58-a48c24514abf" containerName="payload-processing"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670604    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:30:41.670626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.670613    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:30:41.675822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.675788    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820376    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820542 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820414    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxg9\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-kube-api-access-zgxg9\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820550    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629e5415-662a-4211-8ec6-a2b32497de0b-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820592    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820610    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820744    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629e5415-662a-4211-8ec6-a2b32497de0b-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820781    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820826    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.820919 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.820905    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.870743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.868911    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"]
Nov 03 12:30:41.928562 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928457    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629e5415-662a-4211-8ec6-a2b32497de0b-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.928562 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928535    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.928863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928567    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.928863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928621    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629e5415-662a-4211-8ec6-a2b32497de0b-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.928863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928654    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.928863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928729    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.928863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928782    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.929152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928873    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.929152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.928906    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxg9\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-kube-api-access-zgxg9\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.931867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.930633    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629e5415-662a-4211-8ec6-a2b32497de0b-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.931867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.930938    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.931867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.931158    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.931867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.931784    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.932443 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.932420    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.943337 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.943303    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.944045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.944027    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629e5415-662a-4211-8ec6-a2b32497de0b-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.981323 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.981302    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxg9\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-kube-api-access-zgxg9\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.982964 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.982937    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:41.991986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:41.991965    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:42.433230 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:42.433195    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:30:42.433618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:42.433590    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:30:42.433939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:42.433919    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:30:42.434186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:42.434167    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"]
Nov 03 12:30:42.631312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:42.631271    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" event={"ID":"629e5415-662a-4211-8ec6-a2b32497de0b","Type":"ContainerStarted","Data":"e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d"}
Nov 03 12:30:42.631563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:42.631540    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" event={"ID":"629e5415-662a-4211-8ec6-a2b32497de0b","Type":"ContainerStarted","Data":"18f3ca1c79720e474c07b7ec5d8345e95991cc2cfc9346c6a9caf6df12277838"}
Nov 03 12:30:42.992540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:42.992391    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:43.993923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:43.993890    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 12:30:43.994307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:43.993942    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 12:30:44.992802 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:44.992759    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 12:30:44.993009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:44.992816    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 12:30:45.992801 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:45.992767    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 12:30:45.993188 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:45.992828    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 12:30:46.999721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:46.995201    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 12:30:46.999721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:46.995389    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 12:30:47.992501 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:47.992465    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 12:30:47.992744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:47.992526    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 12:30:48.993021 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:48.992991    4909 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 12:30:48.993420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:48.993079    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 12:30:50.035237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:50.035194    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:50.036126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:50.036096    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:50.037188 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:50.037166    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:30:50.192787 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:50.192741    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podStartSLOduration=9.192728125 podStartE2EDuration="9.192728125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:42.745448108 +0000 UTC m=+2391.757914783" watchObservedRunningTime="2026-11-03 12:30:50.192728125 +0000 UTC m=+2399.205194785"
Nov 03 12:30:50.604350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:50.604318    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"]
Nov 03 12:30:50.604573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:50.604547    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48" podUID="27c8a7f1-2405-44d9-a7dc-38ce7e68c122" containerName="maas-api" containerID="cri-o://0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb" gracePeriod=30
Nov 03 12:30:50.903773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:50.903746    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:30:51.006317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.006271    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-openshift-service-ca\") pod \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") "
Nov 03 12:30:51.007159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.006898    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-openshift-service-ca" pod "27c8a7f1-2405-44d9-a7dc-38ce7e68c122" (UID: "27c8a7f1-2405-44d9-a7dc-38ce7e68c122"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:51.007456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.007437    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-kube-api-access-vd8kq\" (UniqueName: \"kubernetes.io/projected/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-kube-api-access-vd8kq\") pod \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") "
Nov 03 12:30:51.008027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.008008    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-metrics-certs\") pod \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") "
Nov 03 12:30:51.008595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.008576    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls\") pod \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\" (UID: \"27c8a7f1-2405-44d9-a7dc-38ce7e68c122\") "
Nov 03 12:30:51.015747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.009793    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:51.023155 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.023101    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-kube-api-access-vd8kq" pod "27c8a7f1-2405-44d9-a7dc-38ce7e68c122" (UID: "27c8a7f1-2405-44d9-a7dc-38ce7e68c122"). InnerVolumeSpecName "kube-api-access-vd8kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:51.043053 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.042971    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls" pod "27c8a7f1-2405-44d9-a7dc-38ce7e68c122" (UID: "27c8a7f1-2405-44d9-a7dc-38ce7e68c122"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:51.043053 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.043000    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-metrics-certs" pod "27c8a7f1-2405-44d9-a7dc-38ce7e68c122" (UID: "27c8a7f1-2405-44d9-a7dc-38ce7e68c122"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:51.118492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.118456    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:51.118492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.118494    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8kq\" (UniqueName: \"kubernetes.io/projected/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-kube-api-access-vd8kq\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:51.118775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.118511    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8a7f1-2405-44d9-a7dc-38ce7e68c122-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:51.423254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.423169    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:30:51.425367 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.425344    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:30:51.721519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.721491    4909 generic.go:356] "Generic (PLEG): container finished" podID="27c8a7f1-2405-44d9-a7dc-38ce7e68c122" containerID="0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb" exitCode=0
Nov 03 12:30:51.721662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.721538    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"
Nov 03 12:30:51.721662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.721579    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48" event={"ID":"27c8a7f1-2405-44d9-a7dc-38ce7e68c122","Type":"ContainerDied","Data":"0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb"}
Nov 03 12:30:51.721662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.721634    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48" event={"ID":"27c8a7f1-2405-44d9-a7dc-38ce7e68c122","Type":"ContainerDied","Data":"b8dfb832436ddcf51e19a3f3963f7c496a3d799afaf5b0ef5bf713a384c34d42"}
Nov 03 12:30:51.721662 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.721660    4909 scope.go:122] "RemoveContainer" containerID="0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb"
Nov 03 12:30:51.731069 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.731053    4909 scope.go:122] "RemoveContainer" containerID="0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb"
Nov 03 12:30:51.731344 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:51.731313    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb\": container with ID starting with 0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb not found: ID does not exist" containerID="0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb"
Nov 03 12:30:51.731398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.731355    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb"} err="failed to get container status \"0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb\": rpc error: code = NotFound desc = could not find container \"0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb\": container with ID starting with 0a8d42e1ef3d218ae79e38d2dc14ba079efe586ddf1ad12c6b0cd63a15d3b5fb not found: ID does not exist"
Nov 03 12:30:51.874089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.874059    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"]
Nov 03 12:30:51.876504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:51.876483    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f3eb2-9747d995d-wll48"]
Nov 03 12:30:53.123067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:53.123034    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c8a7f1-2405-44d9-a7dc-38ce7e68c122" path="/var/lib/kubelet/pods/27c8a7f1-2405-44d9-a7dc-38ce7e68c122/volumes"
Nov 03 12:30:57.237553 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.237517    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"]
Nov 03 12:30:57.238025 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.237769    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q" podUID="d2b92641-53e0-43b5-9e0b-6de2831d67d7" containerName="maas-api" containerID="cri-o://690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7" gracePeriod=30
Nov 03 12:30:57.551496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.551471    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:30:57.686279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.685156    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls\") pod \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") "
Nov 03 12:30:57.686279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.685206    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b92641-53e0-43b5-9e0b-6de2831d67d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b92641-53e0-43b5-9e0b-6de2831d67d7-openshift-service-ca\") pod \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") "
Nov 03 12:30:57.686279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.685275    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-metrics-certs\") pod \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") "
Nov 03 12:30:57.686279 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.685450    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b92641-53e0-43b5-9e0b-6de2831d67d7-kube-api-access-jl687\" (UniqueName: \"kubernetes.io/projected/d2b92641-53e0-43b5-9e0b-6de2831d67d7-kube-api-access-jl687\") pod \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\" (UID: \"d2b92641-53e0-43b5-9e0b-6de2831d67d7\") "
Nov 03 12:30:57.713755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.698930    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b92641-53e0-43b5-9e0b-6de2831d67d7-openshift-service-ca" pod "d2b92641-53e0-43b5-9e0b-6de2831d67d7" (UID: "d2b92641-53e0-43b5-9e0b-6de2831d67d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:57.716158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.716100    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-metrics-certs" pod "d2b92641-53e0-43b5-9e0b-6de2831d67d7" (UID: "d2b92641-53e0-43b5-9e0b-6de2831d67d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:57.716158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.716112    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls" pod "d2b92641-53e0-43b5-9e0b-6de2831d67d7" (UID: "d2b92641-53e0-43b5-9e0b-6de2831d67d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:57.716367 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.716198    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b92641-53e0-43b5-9e0b-6de2831d67d7-kube-api-access-jl687" pod "d2b92641-53e0-43b5-9e0b-6de2831d67d7" (UID: "d2b92641-53e0-43b5-9e0b-6de2831d67d7"). InnerVolumeSpecName "kube-api-access-jl687". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:57.764027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.763993    4909 generic.go:356] "Generic (PLEG): container finished" podID="d2b92641-53e0-43b5-9e0b-6de2831d67d7" containerID="690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7" exitCode=0
Nov 03 12:30:57.764185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.764057    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q" event={"ID":"d2b92641-53e0-43b5-9e0b-6de2831d67d7","Type":"ContainerDied","Data":"690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7"}
Nov 03 12:30:57.764185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.764121    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q" event={"ID":"d2b92641-53e0-43b5-9e0b-6de2831d67d7","Type":"ContainerDied","Data":"ed719293f47a037d2d55067c3b60c5094fdeb0cdedaa5c8304390b44c9732e69"}
Nov 03 12:30:57.764185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.764127    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"
Nov 03 12:30:57.764185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.764144    4909 scope.go:122] "RemoveContainer" containerID="690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7"
Nov 03 12:30:57.775496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.775459    4909 scope.go:122] "RemoveContainer" containerID="690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7"
Nov 03 12:30:57.775884 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:30:57.775847    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7\": container with ID starting with 690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7 not found: ID does not exist" containerID="690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7"
Nov 03 12:30:57.775997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.775892    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7"} err="failed to get container status \"690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7\": rpc error: code = NotFound desc = could not find container \"690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7\": container with ID starting with 690d1a444a52dfe5e6a637481c31ef7589d2b80c33f716bfe0571b3bb3779ac7 not found: ID does not exist"
Nov 03 12:30:57.804502 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.804463    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl687\" (UniqueName: \"kubernetes.io/projected/d2b92641-53e0-43b5-9e0b-6de2831d67d7-kube-api-access-jl687\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:57.804726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.804507    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:57.804726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.804525    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b92641-53e0-43b5-9e0b-6de2831d67d7-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:57.804726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.804541    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b92641-53e0-43b5-9e0b-6de2831d67d7-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:57.919014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.918980    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"]
Nov 03 12:30:57.921287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:57.921262    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04d73d-64cb699769-27h9q"]
Nov 03 12:30:58.744263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:58.744232    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"]
Nov 03 12:30:58.744688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:58.744484    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy" containerID="cri-o://5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a" gracePeriod=30
Nov 03 12:30:59.124828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:30:59.124757    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b92641-53e0-43b5-9e0b-6de2831d67d7" path="/var/lib/kubelet/pods/d2b92641-53e0-43b5-9e0b-6de2831d67d7/volumes"
Nov 03 12:31:00.133050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:00.133008    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:31:04.068263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.068231    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101065    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-podinfo\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101185    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-data\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101252    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-envoy\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101291    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-certs\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101328    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-socket\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101361    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-kube-api-access-jzmjm\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-kube-api-access-jzmjm\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101407    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istiod-ca-cert\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101444    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-credential-socket\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.101742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.101550    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-token\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-token\") pod \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\" (UID: \"c91fdb8b-3546-4258-9068-74b6b5a7c2d8\") "
Nov 03 12:31:04.103995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.103149    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istiod-ca-cert" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:04.103995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.103435    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-socket" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:04.103995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.103680    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-data" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:04.103995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.103841    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-certs" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:04.104581 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.104558    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-credential-socket" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:04.116816 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.116770    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-podinfo" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:31:04.125597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.125551    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-token" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:04.126916 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.126876    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-envoy" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:04.127445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.127412    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-kube-api-access-jzmjm" pod "c91fdb8b-3546-4258-9068-74b6b5a7c2d8" (UID: "c91fdb8b-3546-4258-9068-74b6b5a7c2d8"). InnerVolumeSpecName "kube-api-access-jzmjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:04.203557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203524    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203551    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203561    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203570    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzmjm\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-kube-api-access-jzmjm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203580    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203589    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203599    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203620    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.203812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.203628    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91fdb8b-3546-4258-9068-74b6b5a7c2d8-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:04.812363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.812324    4909 generic.go:356] "Generic (PLEG): container finished" podID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerID="5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a" exitCode=0
Nov 03 12:31:04.812528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.812368    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"
Nov 03 12:31:04.812528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.812377    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" event={"ID":"c91fdb8b-3546-4258-9068-74b6b5a7c2d8","Type":"ContainerDied","Data":"5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a"}
Nov 03 12:31:04.812528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.812444    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb" event={"ID":"c91fdb8b-3546-4258-9068-74b6b5a7c2d8","Type":"ContainerDied","Data":"df76582eea9c57ff8cbc497afb646fc5179994408168e57b1e27cb9c06c52efd"}
Nov 03 12:31:04.812528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.812471    4909 scope.go:122] "RemoveContainer" containerID="5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a"
Nov 03 12:31:04.823033 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.823015    4909 scope.go:122] "RemoveContainer" containerID="04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058"
Nov 03 12:31:04.831411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.831388    4909 scope.go:122] "RemoveContainer" containerID="5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a"
Nov 03 12:31:04.831671 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:31:04.831648    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a\": container with ID starting with 5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a not found: ID does not exist" containerID="5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a"
Nov 03 12:31:04.831752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.831681    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a"} err="failed to get container status \"5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a\": rpc error: code = NotFound desc = could not find container \"5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a\": container with ID starting with 5ae595b1d3edd1114a70dd3bbba4e4f798115016a4e4e3422b43cf2bede0a65a not found: ID does not exist"
Nov 03 12:31:04.831752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.831714    4909 scope.go:122] "RemoveContainer" containerID="04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058"
Nov 03 12:31:04.831960 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:31:04.831940    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058\": container with ID starting with 04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058 not found: ID does not exist" containerID="04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058"
Nov 03 12:31:04.832005 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.831967    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058"} err="failed to get container status \"04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058\": rpc error: code = NotFound desc = could not find container \"04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058\": container with ID starting with 04c3e5fe56afa0e7277b8372a3f0930b0b17c018187721c017aa5d0aebe08058 not found: ID does not exist"
Nov 03 12:31:04.967208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.967170    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"]
Nov 03 12:31:04.969882 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:04.969855    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-40ffcd-openshift-default-6bc5fbc5b5-fpvzb"]
Nov 03 12:31:05.123251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:05.123181    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" path="/var/lib/kubelet/pods/c91fdb8b-3546-4258-9068-74b6b5a7c2d8/volumes"
Nov 03 12:31:09.563795 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:31:09.563754    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:31:10.848324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:10.848292    4909 generic.go:356] "Generic (PLEG): container finished" podID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerID="18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4" exitCode=0
Nov 03 12:31:10.848830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:10.848363    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" event={"ID":"04b72109-dd79-4824-a3de-1c1b87f520f8","Type":"ContainerDied","Data":"18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4"}
Nov 03 12:31:10.848830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:10.848770    4909 scope.go:122] "RemoveContainer" containerID="18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4"
Nov 03 12:31:10.850152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:10.850128    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:10.850210 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:10.850190    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:10.850247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:10.850223    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:11.867357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:11.867313    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" event={"ID":"04b72109-dd79-4824-a3de-1c1b87f520f8","Type":"ContainerStarted","Data":"e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a"}
Nov 03 12:31:11.962999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:11.961807    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:31:12.224843 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:12.224810    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:31:12.225750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:12.225721    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 12:31:12.225871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:12.225775    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 12:31:13.224826 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:13.224785    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 12:31:13.225306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:13.224844    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 12:31:14.224880 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:14.224844    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 12:31:14.225363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:14.224911    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 12:31:15.225524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:15.225489    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 12:31:15.225942 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:15.225547    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 12:31:16.225349 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:16.225317    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 12:31:16.225512 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:16.225367    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 12:31:17.228970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:17.228940    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:31:17.229905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:17.229885    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:31:23.909390 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:23.909359    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"]
Nov 03 12:31:23.909858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:23.909577    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy" containerID="cri-o://bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f" gracePeriod=30
Nov 03 12:31:26.004589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.004556    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"]
Nov 03 12:31:26.005045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005031    4909 cpu_manager.go:414] "containerMap: removing container" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:31:26.005045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005045    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005062    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d2b92641-53e0-43b5-9e0b-6de2831d67d7" containerName="maas-api"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005067    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b92641-53e0-43b5-9e0b-6de2831d67d7" containerName="maas-api"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005079    4909 cpu_manager.go:414] "containerMap: removing container" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005085    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005112    4909 cpu_manager.go:414] "containerMap: removing container" podUID="27c8a7f1-2405-44d9-a7dc-38ce7e68c122" containerName="maas-api"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005117    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c8a7f1-2405-44d9-a7dc-38ce7e68c122" containerName="maas-api"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005129    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy"
Nov 03 12:31:26.005134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005134    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005139    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005145    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005213    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="59812dc6-823b-4b54-acef-1b7c1fccb72c" containerName="cleanup"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005226    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c8a7f1-2405-44d9-a7dc-38ce7e68c122" containerName="maas-api"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005233    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005238    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91fdb8b-3546-4258-9068-74b6b5a7c2d8" containerName="istio-proxy"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005245    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b76a780-e28f-4123-8cff-5435c2d959f3" containerName="cleanup"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005253    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b92641-53e0-43b5-9e0b-6de2831d67d7" containerName="maas-api"
Nov 03 12:31:26.005409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.005259    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="19130dfa-2a11-4bde-b1f4-97536999d294" containerName="cleanup"
Nov 03 12:31:26.008826 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.008805    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.017879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.017841    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-socket\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.017995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.017900    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-token\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.018073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.018034    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-data\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.018134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.018121    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b95018a-6308-4bb1-b42d-3ac6f472a936-istiod-ca-cert\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.018195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.018148    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-credential-socket\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.018195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.018183    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-envoy\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.018325 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.018305    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-certs\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.018386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.018338    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnsl\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-kube-api-access-2tnsl\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.018444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.018396    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-podinfo\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.070254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.070224    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"]
Nov 03 12:31:26.103839 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.103807    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-eb4a0f-openshift-default-dockercfg-5cdgz\""
Nov 03 12:31:26.119199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-socket\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119221    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-token\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119248    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-data\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119277    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b95018a-6308-4bb1-b42d-3ac6f472a936-istiod-ca-cert\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119295    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-credential-socket\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119341    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-envoy\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119427    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-certs\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119463    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnsl\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-kube-api-access-2tnsl\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119488    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-podinfo\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119870 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119626    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-socket\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119870 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119678    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-data\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.119870 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.119801    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-credential-socket\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.120050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.120014    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-certs\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.120241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.120222    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b95018a-6308-4bb1-b42d-3ac6f472a936-istiod-ca-cert\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.122078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.122059    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-envoy\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.122290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.122270    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-podinfo\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.182722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.182231    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-token\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.188207 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.188083    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnsl\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-kube-api-access-2tnsl\") pod \"e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") " pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.322992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.322914    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:26.738521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.738482    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"]
Nov 03 12:31:26.774658 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:31:26.774631    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b95018a_6308_4bb1_b42d_3ac6f472a936.slice/crio-9d459acb39c38a4735b9e097146636bc2b6b95413818f16100c789083385a1e5 WatchSource:0}: Error finding container 9d459acb39c38a4735b9e097146636bc2b6b95413818f16100c789083385a1e5: Status 404 returned error can't find the container with id 9d459acb39c38a4735b9e097146636bc2b6b95413818f16100c789083385a1e5
Nov 03 12:31:26.776947 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.776912    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:26.777030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.776979    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:26.777030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.777024    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:26.949706 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.949661    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" event={"ID":"9b95018a-6308-4bb1-b42d-3ac6f472a936","Type":"ContainerStarted","Data":"b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651"}
Nov 03 12:31:26.949873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:26.949736    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" event={"ID":"9b95018a-6308-4bb1-b42d-3ac6f472a936","Type":"ContainerStarted","Data":"9d459acb39c38a4735b9e097146636bc2b6b95413818f16100c789083385a1e5"}
Nov 03 12:31:27.058114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:27.058013    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podStartSLOduration=2.057997936 podStartE2EDuration="2.057997936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:27.056482625 +0000 UTC m=+2436.068949286" watchObservedRunningTime="2026-11-03 12:31:27.057997936 +0000 UTC m=+2436.070464599"
Nov 03 12:31:27.229015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:27.228981    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:31:27.229181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:27.229037    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:31:27.323372 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:27.323283    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:27.328376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:27.328354    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:27.955985 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:27.955952    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:27.956948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:27.956929    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:31:28.838574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:28.838541    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:31:29.171098 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.171075    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:31:29.253637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253611    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-token\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-token\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.253775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253641    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-credential-socket\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.253775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253726    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-data\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.253775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253763    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-certs\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.253937 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253876    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-envoy\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.253937 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253905    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-socket\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.254046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253944    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-podinfo\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.254046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.253978    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-kube-api-access-bn4bj\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-kube-api-access-bn4bj\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.254046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254028    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istiod-ca-cert\") pod \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\" (UID: \"f3e6b8e3-35ed-4714-b43e-a63d7267daa2\") "
Nov 03 12:31:29.254046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254031    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-credential-socket" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:29.254260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254036    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-data" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:29.254426 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254345    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istiod-ca-cert" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:29.254426 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254381    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-certs" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:29.254603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254513    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-socket" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:29.254603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254572    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.254603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254593    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.254792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254608    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.254792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.254624    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.256226 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.256201    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-token" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:29.256333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.256276    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-podinfo" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:31:29.256486 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.256470    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-envoy" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:29.256609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.256596    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-kube-api-access-bn4bj" pod "f3e6b8e3-35ed-4714-b43e-a63d7267daa2" (UID: "f3e6b8e3-35ed-4714-b43e-a63d7267daa2"). InnerVolumeSpecName "kube-api-access-bn4bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:29.355920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.355886    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.356048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.355922    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.356048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.355937    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.356048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.355952    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4bj\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-kube-api-access-bn4bj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.356048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.355967    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e6b8e3-35ed-4714-b43e-a63d7267daa2-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:29.974525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.974488    4909 generic.go:356] "Generic (PLEG): container finished" podID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerID="bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f" exitCode=0
Nov 03 12:31:29.975088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.974556    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"
Nov 03 12:31:29.975088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.974634    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" event={"ID":"f3e6b8e3-35ed-4714-b43e-a63d7267daa2","Type":"ContainerDied","Data":"bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f"}
Nov 03 12:31:29.975088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.974684    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57" event={"ID":"f3e6b8e3-35ed-4714-b43e-a63d7267daa2","Type":"ContainerDied","Data":"cbc4f7fffa1b7e22c25beaf5038482db0a6a3b08d9d12cfce282a34668966676"}
Nov 03 12:31:29.975088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.974738    4909 scope.go:122] "RemoveContainer" containerID="bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f"
Nov 03 12:31:29.998054 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:29.998024    4909 scope.go:122] "RemoveContainer" containerID="d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f"
Nov 03 12:31:30.013963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:30.013847    4909 scope.go:122] "RemoveContainer" containerID="bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f"
Nov 03 12:31:30.014552 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:31:30.014387    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f\": container with ID starting with bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f not found: ID does not exist" containerID="bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f"
Nov 03 12:31:30.014552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:30.014429    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f"} err="failed to get container status \"bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f\": rpc error: code = NotFound desc = could not find container \"bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f\": container with ID starting with bd8c5cfb69828db5badcf5504766663984dcf4c1f8204e5627232d1a5379e81f not found: ID does not exist"
Nov 03 12:31:30.014552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:30.014456    4909 scope.go:122] "RemoveContainer" containerID="d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f"
Nov 03 12:31:30.015386 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:31:30.015282    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f\": container with ID starting with d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f not found: ID does not exist" containerID="d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f"
Nov 03 12:31:30.015386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:30.015323    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f"} err="failed to get container status \"d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f\": rpc error: code = NotFound desc = could not find container \"d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f\": container with ID starting with d9ce946cb5cf03db34859b0417122589a87ecc7fe8689fe68cb7c54ef005367f not found: ID does not exist"
Nov 03 12:31:30.204720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:30.199762    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"]
Nov 03 12:31:30.204720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:30.199830    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d25504-openshift-default-6cfff4c68f-mvk57"]
Nov 03 12:31:31.204491 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:31.204452    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" path="/var/lib/kubelet/pods/f3e6b8e3-35ed-4714-b43e-a63d7267daa2/volumes"
Nov 03 12:31:36.147645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.147612    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"]
Nov 03 12:31:36.148161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.148091    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy"
Nov 03 12:31:36.148161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.148102    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy"
Nov 03 12:31:36.148161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.148126    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy"
Nov 03 12:31:36.148161 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.148132    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy"
Nov 03 12:31:36.148320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.148220    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy"
Nov 03 12:31:36.148320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.148231    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e6b8e3-35ed-4714-b43e-a63d7267daa2" containerName="istio-proxy"
Nov 03 12:31:36.152881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.152864    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.204349 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.204321    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"]
Nov 03 12:31:36.307885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.307829    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-eb4a0f-dockercfg-mm6r8\""
Nov 03 12:31:36.308092 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.307876    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-eb4a0f\""
Nov 03 12:31:36.309736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.308575    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8f3341-736d-4961-bb21-354243728c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.309736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.308872    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxg4\" (UniqueName: \"kubernetes.io/projected/fb8f3341-736d-4961-bb21-354243728c5b-kube-api-access-tsxg4\") pod \"payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.318635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.317385    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"]
Nov 03 12:31:36.329448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.326297    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:36.369083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.369018    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"]
Nov 03 12:31:36.410598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.410533    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxg4\" (UniqueName: \"kubernetes.io/projected/fb8f3341-736d-4961-bb21-354243728c5b-kube-api-access-tsxg4\") pod \"payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.410734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.410630    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8f3341-736d-4961-bb21-354243728c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.411296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.411277    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8f3341-736d-4961-bb21-354243728c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.465097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.465074    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxg4\" (UniqueName: \"kubernetes.io/projected/fb8f3341-736d-4961-bb21-354243728c5b-kube-api-access-tsxg4\") pod \"payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.465884 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.465863    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:36.512627 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.512588    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57674fbd-3ec2-4612-9abc-0145666d1fd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:36.512777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.512755    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf69\" (UniqueName: \"kubernetes.io/projected/57674fbd-3ec2-4612-9abc-0145666d1fd2-kube-api-access-vpf69\") pod \"payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:36.614353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.614307    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57674fbd-3ec2-4612-9abc-0145666d1fd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:36.614538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.614385    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf69\" (UniqueName: \"kubernetes.io/projected/57674fbd-3ec2-4612-9abc-0145666d1fd2-kube-api-access-vpf69\") pod \"payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:36.615293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.615261    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57674fbd-3ec2-4612-9abc-0145666d1fd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:36.668608 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.668546    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf69\" (UniqueName: \"kubernetes.io/projected/57674fbd-3ec2-4612-9abc-0145666d1fd2-kube-api-access-vpf69\") pod \"payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:36.828414 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.828390    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"]
Nov 03 12:31:36.830225 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:31:36.830197    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8f3341_736d_4961_bb21_354243728c5b.slice/crio-9d86b01bcf8a6c8fb2ef50b41dd922ffee73b1872a104f69e58da4a025b6123d WatchSource:0}: Error finding container 9d86b01bcf8a6c8fb2ef50b41dd922ffee73b1872a104f69e58da4a025b6123d: Status 404 returned error can't find the container with id 9d86b01bcf8a6c8fb2ef50b41dd922ffee73b1872a104f69e58da4a025b6123d
Nov 03 12:31:36.953914 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:36.953891    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:37.135833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:37.135793    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2" event={"ID":"fb8f3341-736d-4961-bb21-354243728c5b","Type":"ContainerStarted","Data":"9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca"}
Nov 03 12:31:37.136010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:37.135902    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2" event={"ID":"fb8f3341-736d-4961-bb21-354243728c5b","Type":"ContainerStarted","Data":"9d86b01bcf8a6c8fb2ef50b41dd922ffee73b1872a104f69e58da4a025b6123d"}
Nov 03 12:31:37.136010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:37.135942    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:37.246416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:37.246316    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2" podStartSLOduration=1.246300006 podStartE2EDuration="1.246300006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:37.244482115 +0000 UTC m=+2446.256948770" watchObservedRunningTime="2026-11-03 12:31:37.246300006 +0000 UTC m=+2446.258766669"
Nov 03 12:31:37.321253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:37.321227    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"]
Nov 03 12:31:37.322194 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:31:37.322167    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57674fbd_3ec2_4612_9abc_0145666d1fd2.slice/crio-9cac1c958bb0b6044bb005ebd0f661dd939256ee6041d9385e8a5935cfbbdfac WatchSource:0}: Error finding container 9cac1c958bb0b6044bb005ebd0f661dd939256ee6041d9385e8a5935cfbbdfac: Status 404 returned error can't find the container with id 9cac1c958bb0b6044bb005ebd0f661dd939256ee6041d9385e8a5935cfbbdfac
Nov 03 12:31:38.142566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:38.142531    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95" event={"ID":"57674fbd-3ec2-4612-9abc-0145666d1fd2","Type":"ContainerStarted","Data":"70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e"}
Nov 03 12:31:38.142759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:38.142583    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95" event={"ID":"57674fbd-3ec2-4612-9abc-0145666d1fd2","Type":"ContainerStarted","Data":"9cac1c958bb0b6044bb005ebd0f661dd939256ee6041d9385e8a5935cfbbdfac"}
Nov 03 12:31:38.247736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:38.247648    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95" podStartSLOduration=2.247631468 podStartE2EDuration="2.247631468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:38.247249491 +0000 UTC m=+2447.259716154" watchObservedRunningTime="2026-11-03 12:31:38.247631468 +0000 UTC m=+2447.260098132"
Nov 03 12:31:39.149776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:39.149738    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:39.564164 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:31:39.564123    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:31:48.145829 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:48.145797    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:31:50.159344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:50.159308    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:31:55.449641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:55.449606    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:31:58.151723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.150849    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"]
Nov 03 12:31:58.158189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.157337    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.232625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.232594    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9893f0-openshift-default-dockercfg-ptlmk\""
Nov 03 12:31:58.253173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.253136    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.253505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.253483    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.253760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.253734    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.253933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.253917    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.254134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.254115    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.254315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.254298    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztrl\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-kube-api-access-2ztrl\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.254484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.254467    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.254649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.254632    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.254846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.254830    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24197277-684e-4545-9b2c-1aaba25a5f3c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.285424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.285389    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"]
Nov 03 12:31:58.361736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.361687    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24197277-684e-4545-9b2c-1aaba25a5f3c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.361992    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362062    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362201 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362167    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362242 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362212    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362295    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362480    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztrl\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-kube-api-access-2ztrl\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362625    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362489    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24197277-684e-4545-9b2c-1aaba25a5f3c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362368    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362564    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362742    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.362788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.362763    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.363172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.363150    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.365024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.365003    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.365339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.365317    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.415106 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.415050    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.415195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.415132    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztrl\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-kube-api-access-2ztrl\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.490354 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.490326    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:58.907877 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.907779    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"]
Nov 03 12:31:58.909962 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:31:58.909935    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24197277_684e_4545_9b2c_1aaba25a5f3c.slice/crio-58967c485f7de6341a705df9819cf95166ab1fa4c0dac7b800179072c054a97c WatchSource:0}: Error finding container 58967c485f7de6341a705df9819cf95166ab1fa4c0dac7b800179072c054a97c: Status 404 returned error can't find the container with id 58967c485f7de6341a705df9819cf95166ab1fa4c0dac7b800179072c054a97c
Nov 03 12:31:58.912321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.912280    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:58.912395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.912351    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:58.912395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:58.912382    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:31:59.310358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:59.310324    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" event={"ID":"24197277-684e-4545-9b2c-1aaba25a5f3c","Type":"ContainerStarted","Data":"d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0"}
Nov 03 12:31:59.310767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:59.310377    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" event={"ID":"24197277-684e-4545-9b2c-1aaba25a5f3c","Type":"ContainerStarted","Data":"58967c485f7de6341a705df9819cf95166ab1fa4c0dac7b800179072c054a97c"}
Nov 03 12:31:59.419046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:59.418994    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podStartSLOduration=1.418980074 podStartE2EDuration="1.418980074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:59.417294108 +0000 UTC m=+2468.429760771" watchObservedRunningTime="2026-11-03 12:31:59.418980074 +0000 UTC m=+2468.431446737"
Nov 03 12:31:59.490760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:59.490732    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:31:59.492098 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:59.492076    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:31:59.492182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:31:59.492134    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:00.491404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:00.491363    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:32:00.491824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:00.491422    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:01.491220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:01.491137    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:32:01.491220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:01.491193    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:02.491550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:02.491513    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:32:02.491940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:02.491571    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:03.491349 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:03.491309    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:32:03.491541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:03.491374    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:04.491714 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:04.491661    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:32:04.492189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:04.491739    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:05.490790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:05.490750    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:32:05.490995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:05.490805    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:06.491112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:06.491077    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:32:06.491502 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:06.491130    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 12:32:07.495840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:07.495804    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:32:07.496248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:07.496224    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:32:07.496909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:07.496886    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:32:09.572306 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:32:09.572248    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:32:22.890674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:22.890641    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:32:25.203722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.202775    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.247723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.246784    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:32:25.258721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.256763    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 03 12:32:25.264807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.264770    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.287835 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.287785    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.336834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.336782    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.687778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.686773    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.906001 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:25.902565    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:26.084829 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.084781    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" podUID="955970d0-cd2b-46b5-8d24-4cf06cf33989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:26.203922 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.203859    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:26.205834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.205811    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:32:26.250731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.249770    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:26.250731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.250163    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:32:26.279298 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.279203    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:26.279593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.279565    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:32:26.291834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.291779    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:26.292454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.292433    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:32:26.419779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.408241    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 12:32:26.561748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.544714    4909 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 03 12:32:26.638721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.615580    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 12:32:26.678776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.676851    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 12:32:26.678942 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:26.678908    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 12:32:27.260823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.260778    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"]
Nov 03 12:32:27.261746 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.261712    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy" containerID="cri-o://62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7" gracePeriod=30
Nov 03 12:32:27.769342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.769304    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"]
Nov 03 12:32:27.776974 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.775939    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:27.825170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.825133    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1250b3\""
Nov 03 12:32:27.842941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.842903    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1250b3-dockercfg-v69jr\""
Nov 03 12:32:27.845146 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.845056    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tl2v\" (UniqueName: \"kubernetes.io/projected/0508f102-e252-43f4-a8a9-58fe6b0a06e3-kube-api-access-6tl2v\") pod \"payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:27.845382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.845357    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0508f102-e252-43f4-a8a9-58fe6b0a06e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:27.867743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.864791    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"]
Nov 03 12:32:27.981032 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.980985    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tl2v\" (UniqueName: \"kubernetes.io/projected/0508f102-e252-43f4-a8a9-58fe6b0a06e3-kube-api-access-6tl2v\") pod \"payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:27.981240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.981064    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0508f102-e252-43f4-a8a9-58fe6b0a06e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:27.982193 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.982165    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0508f102-e252-43f4-a8a9-58fe6b0a06e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:27.983915 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.983888    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"]
Nov 03 12:32:27.989133 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.989111    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:27.991764 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:27.991741    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"]
Nov 03 12:32:28.035780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.035712    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tl2v\" (UniqueName: \"kubernetes.io/projected/0508f102-e252-43f4-a8a9-58fe6b0a06e3-kube-api-access-6tl2v\") pod \"payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:28.082293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.082259    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6ct\" (UniqueName: \"kubernetes.io/projected/ded63b27-8f76-41d1-a992-38982f6a69b4-kube-api-access-vw6ct\") pod \"payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:28.082467 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.082315    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded63b27-8f76-41d1-a992-38982f6a69b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:28.088360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.088333    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"]
Nov 03 12:32:28.092583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.092561    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.093370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.093349    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:28.177173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.174908    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ab27b8-openshift-default-dockercfg-l9952\""
Nov 03 12:32:28.177173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.175978    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"]
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184018    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396f2942-0950-46ad-812b-d506efc6bfbd-istiod-ca-cert\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184058    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-socket\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184095    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-data\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184120    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-istio-token\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184158    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9mt\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-kube-api-access-ls9mt\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184190    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-credential-socket\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184207    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-envoy\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184232    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6ct\" (UniqueName: \"kubernetes.io/projected/ded63b27-8f76-41d1-a992-38982f6a69b4-kube-api-access-vw6ct\") pod \"payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184263    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded63b27-8f76-41d1-a992-38982f6a69b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184291    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-certs\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.184824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.184339    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396f2942-0950-46ad-812b-d506efc6bfbd-istio-podinfo\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.187540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.185254    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded63b27-8f76-41d1-a992-38982f6a69b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:28.266349 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.266278    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6ct\" (UniqueName: \"kubernetes.io/projected/ded63b27-8f76-41d1-a992-38982f6a69b4-kube-api-access-vw6ct\") pod \"payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:28.285715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.285671    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-socket\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.285912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.285757    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-data\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.285912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.285801    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-istio-token\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.285912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.285866    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9mt\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-kube-api-access-ls9mt\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.285912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.285899    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-credential-socket\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.285928    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-envoy\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.286006    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-certs\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.286084    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396f2942-0950-46ad-812b-d506efc6bfbd-istio-podinfo\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.286093    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-socket\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.286145    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-data\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.286206    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396f2942-0950-46ad-812b-d506efc6bfbd-istiod-ca-cert\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.286413    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-credential-socket\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.286933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.286799    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-certs\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.287113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.287093    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396f2942-0950-46ad-812b-d506efc6bfbd-istiod-ca-cert\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.295739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.288841    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396f2942-0950-46ad-812b-d506efc6bfbd-istio-podinfo\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.295739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.289055    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-envoy\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.304873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.304849    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:28.364356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.364287    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-istio-token\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.365219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.365186    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9mt\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-kube-api-access-ls9mt\") pod \"e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") " pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.448736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.426898    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:28.500412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.500369    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"]
Nov 03 12:32:28.509968 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:28.509934    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0508f102_e252_43f4_a8a9_58fe6b0a06e3.slice/crio-51586126d13a04f7e55bc96ab06d7f9f0b0a9786c8ea60731463cfc775dfb6fb WatchSource:0}: Error finding container 51586126d13a04f7e55bc96ab06d7f9f0b0a9786c8ea60731463cfc775dfb6fb: Status 404 returned error can't find the container with id 51586126d13a04f7e55bc96ab06d7f9f0b0a9786c8ea60731463cfc775dfb6fb
Nov 03 12:32:28.685275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.685000    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" event={"ID":"0508f102-e252-43f4-a8a9-58fe6b0a06e3","Type":"ContainerStarted","Data":"8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962"}
Nov 03 12:32:28.685275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.685066    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" event={"ID":"0508f102-e252-43f4-a8a9-58fe6b0a06e3","Type":"ContainerStarted","Data":"51586126d13a04f7e55bc96ab06d7f9f0b0a9786c8ea60731463cfc775dfb6fb"}
Nov 03 12:32:28.685275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.685212    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:28.696171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.696138    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"]
Nov 03 12:32:28.697925 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:28.697902    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded63b27_8f76_41d1_a992_38982f6a69b4.slice/crio-afeb623d1fc9e4ad19f1caa53c083d155ecd9329a8c8f67ddccd890d25288bf1 WatchSource:0}: Error finding container afeb623d1fc9e4ad19f1caa53c083d155ecd9329a8c8f67ddccd890d25288bf1: Status 404 returned error can't find the container with id afeb623d1fc9e4ad19f1caa53c083d155ecd9329a8c8f67ddccd890d25288bf1
Nov 03 12:32:28.791465 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.791416    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" podStartSLOduration=1.79140241 podStartE2EDuration="1.79140241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:28.788864183 +0000 UTC m=+2497.801330847" watchObservedRunningTime="2026-11-03 12:32:28.79140241 +0000 UTC m=+2497.803869073"
Nov 03 12:32:28.891308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.891278    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"]
Nov 03 12:32:28.893627 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:28.893596    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396f2942_0950_46ad_812b_d506efc6bfbd.slice/crio-e651538e4689cfaa022131e079a8730667162ef91331bfc99d6ca7e3ff041684 WatchSource:0}: Error finding container e651538e4689cfaa022131e079a8730667162ef91331bfc99d6ca7e3ff041684: Status 404 returned error can't find the container with id e651538e4689cfaa022131e079a8730667162ef91331bfc99d6ca7e3ff041684
Nov 03 12:32:28.896313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.896279    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:32:28.896427 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.896347    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:32:28.896427 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:28.896392    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:32:29.692184 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:29.692137    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" event={"ID":"396f2942-0950-46ad-812b-d506efc6bfbd","Type":"ContainerStarted","Data":"bf75a593aa5f87ca9835edeb6baad3311b9567c931adab7b63caf4512bd5469d"}
Nov 03 12:32:29.692665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:29.692197    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" event={"ID":"396f2942-0950-46ad-812b-d506efc6bfbd","Type":"ContainerStarted","Data":"e651538e4689cfaa022131e079a8730667162ef91331bfc99d6ca7e3ff041684"}
Nov 03 12:32:29.693981 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:29.693953    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" event={"ID":"ded63b27-8f76-41d1-a992-38982f6a69b4","Type":"ContainerStarted","Data":"bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc"}
Nov 03 12:32:29.694102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:29.694006    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:29.694102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:29.694023    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" event={"ID":"ded63b27-8f76-41d1-a992-38982f6a69b4","Type":"ContainerStarted","Data":"afeb623d1fc9e4ad19f1caa53c083d155ecd9329a8c8f67ddccd890d25288bf1"}
Nov 03 12:32:29.816459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:29.816384    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podStartSLOduration=1.816369828 podStartE2EDuration="1.816369828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:29.815971808 +0000 UTC m=+2498.828438468" watchObservedRunningTime="2026-11-03 12:32:29.816369828 +0000 UTC m=+2498.828836492"
Nov 03 12:32:29.920991 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:29.920892    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" podStartSLOduration=2.9208760910000002 podStartE2EDuration="2.920876091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:29.920111082 +0000 UTC m=+2498.932577747" watchObservedRunningTime="2026-11-03 12:32:29.920876091 +0000 UTC m=+2498.933342755"
Nov 03 12:32:30.447565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:30.447533    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:30.453452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:30.453429    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:30.704904 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:30.704875    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:30.705898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:30.705873    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:32:32.558982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.558906    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:32:32.637356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637323    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-token\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-token\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637396    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-podinfo\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637486    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-certs\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637553    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-credential-socket\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637588    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-data\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637617    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-envoy\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637665    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-kube-api-access-c8272\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-kube-api-access-c8272\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637812    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-socket\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.637859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637841    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-certs" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:32.637859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637848    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istiod-ca-cert\") pod \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\" (UID: \"4c6815b8-75a6-4b9c-bc01-aac63d707d94\") "
Nov 03 12:32:32.638102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637864    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-credential-socket" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:32.638102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.637976    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-data" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:32.638102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.638059    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-socket" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:32.638285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.638266    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istiod-ca-cert" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:32.638423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.638406    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.638497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.638427    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.638497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.638441    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.638497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.638449    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.638497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.638460    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.639941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.639918    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-envoy" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:32.639941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.639916    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-kube-api-access-c8272" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "kube-api-access-c8272". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:32.640116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.640061    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-podinfo" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:32:32.641037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.641007    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-token" pod "4c6815b8-75a6-4b9c-bc01-aac63d707d94" (UID: "4c6815b8-75a6-4b9c-bc01-aac63d707d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:32.717823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.717794    4909 generic.go:356] "Generic (PLEG): container finished" podID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerID="62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7" exitCode=0
Nov 03 12:32:32.717967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.717863    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"
Nov 03 12:32:32.717967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.717883    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" event={"ID":"4c6815b8-75a6-4b9c-bc01-aac63d707d94","Type":"ContainerDied","Data":"62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7"}
Nov 03 12:32:32.717967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.717947    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7" event={"ID":"4c6815b8-75a6-4b9c-bc01-aac63d707d94","Type":"ContainerDied","Data":"6914831d8433881ede31c0c3e4fd62155aefeeb1a46c8dfd8c97dc92909fbc44"}
Nov 03 12:32:32.718116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.717978    4909 scope.go:122] "RemoveContainer" containerID="62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7"
Nov 03 12:32:32.731638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.731615    4909 scope.go:122] "RemoveContainer" containerID="cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b"
Nov 03 12:32:32.739848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.739428    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.739966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.739860    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.739966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.739898    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6815b8-75a6-4b9c-bc01-aac63d707d94-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.740071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.739967    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8272\" (UniqueName: \"kubernetes.io/projected/4c6815b8-75a6-4b9c-bc01-aac63d707d94-kube-api-access-c8272\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:32.742997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.742976    4909 scope.go:122] "RemoveContainer" containerID="62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7"
Nov 03 12:32:32.743258 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:32:32.743232    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7\": container with ID starting with 62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7 not found: ID does not exist" containerID="62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7"
Nov 03 12:32:32.743353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.743268    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7"} err="failed to get container status \"62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7\": rpc error: code = NotFound desc = could not find container \"62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7\": container with ID starting with 62adade82fd339874a5bb6290e944a2b6d9ff69522602f4e9fbff92b501b1bd7 not found: ID does not exist"
Nov 03 12:32:32.743353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.743291    4909 scope.go:122] "RemoveContainer" containerID="cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b"
Nov 03 12:32:32.743559 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:32:32.743536    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b\": container with ID starting with cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b not found: ID does not exist" containerID="cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b"
Nov 03 12:32:32.743625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.743566    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b"} err="failed to get container status \"cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b\": rpc error: code = NotFound desc = could not find container \"cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b\": container with ID starting with cd6cfad5bb67157af7a34e8527a620155cfe9b2be9a03534348801151ba5124b not found: ID does not exist"
Nov 03 12:32:32.873087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.873004    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"]
Nov 03 12:32:32.875832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:32.875803    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-gbcm7"]
Nov 03 12:32:33.123634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:33.123562    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" path="/var/lib/kubelet/pods/4c6815b8-75a6-4b9c-bc01-aac63d707d94/volumes"
Nov 03 12:32:33.724264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:33.724237    4909 generic.go:356] "Generic (PLEG): container finished" podID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerID="b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651" exitCode=0
Nov 03 12:32:33.724626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:33.724310    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" event={"ID":"9b95018a-6308-4bb1-b42d-3ac6f472a936","Type":"ContainerDied","Data":"b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651"}
Nov 03 12:32:33.724800 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:33.724784    4909 scope.go:122] "RemoveContainer" containerID="b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651"
Nov 03 12:32:33.728997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:33.727085    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:32:33.728997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:33.727257    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:32:33.728997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:33.727475    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:32:34.737296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:34.737252    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" event={"ID":"9b95018a-6308-4bb1-b42d-3ac6f472a936","Type":"ContainerStarted","Data":"880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f"}
Nov 03 12:32:35.323725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:35.323670    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:32:35.323908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:35.323860    4909 patch_prober.go:29] interesting pod/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 12:32:35.323956 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:35.323911    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 12:32:36.098998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.098965    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"]
Nov 03 12:32:36.099687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.099665    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy"
Nov 03 12:32:36.099761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.099689    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy"
Nov 03 12:32:36.099761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.099737    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy"
Nov 03 12:32:36.099761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.099743    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy"
Nov 03 12:32:36.099862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.099833    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy"
Nov 03 12:32:36.099862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.099846    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6815b8-75a6-4b9c-bc01-aac63d707d94" containerName="istio-proxy"
Nov 03 12:32:36.102524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.102505    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:36.214720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.213985    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ab27b8\""
Nov 03 12:32:36.214720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.214103    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ab27b8-dockercfg-pcchw\""
Nov 03 12:32:36.237986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.237079    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplbj\" (UniqueName: \"kubernetes.io/projected/c8c320c1-862e-4421-bd97-f3e64bd88d4d-kube-api-access-zplbj\") pod \"payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:36.237986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.237172    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c320c1-862e-4421-bd97-f3e64bd88d4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:36.305719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.303225    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"]
Nov 03 12:32:36.310723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.310658    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"]
Nov 03 12:32:36.313797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.313738    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.323797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.323773    4909 patch_prober.go:29] interesting pod/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 12:32:36.323911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.323817    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 12:32:36.326762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.326744    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"]
Nov 03 12:32:36.364000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.363909    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplbj\" (UniqueName: \"kubernetes.io/projected/c8c320c1-862e-4421-bd97-f3e64bd88d4d-kube-api-access-zplbj\") pod \"payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:36.364000 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.363969    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c320c1-862e-4421-bd97-f3e64bd88d4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:36.364820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.364794    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c320c1-862e-4421-bd97-f3e64bd88d4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:36.416673 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.416641    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplbj\" (UniqueName: \"kubernetes.io/projected/c8c320c1-862e-4421-bd97-f3e64bd88d4d-kube-api-access-zplbj\") pod \"payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:36.465294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.465248    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqn4h\" (UniqueName: \"kubernetes.io/projected/71ec8b30-32ec-407b-ad71-25dfdde68e79-kube-api-access-dqn4h\") pod \"payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.465584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.465555    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec8b30-32ec-407b-ad71-25dfdde68e79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.566528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.566493    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec8b30-32ec-407b-ad71-25dfdde68e79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.566717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.566572    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqn4h\" (UniqueName: \"kubernetes.io/projected/71ec8b30-32ec-407b-ad71-25dfdde68e79-kube-api-access-dqn4h\") pod \"payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.567304 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.567283    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec8b30-32ec-407b-ad71-25dfdde68e79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.619409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.619323    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqn4h\" (UniqueName: \"kubernetes.io/projected/71ec8b30-32ec-407b-ad71-25dfdde68e79-kube-api-access-dqn4h\") pod \"payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.632177 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.632145    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:36.717107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:36.716922    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:37.031618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.031592    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"]
Nov 03 12:32:37.032135 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:37.032109    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ec8b30_32ec_407b_ad71_25dfdde68e79.slice/crio-c5e7b75ddafef2c7bb94b9266bfed1d7ab0d289b917b70e5e95a3684920b51ae WatchSource:0}: Error finding container c5e7b75ddafef2c7bb94b9266bfed1d7ab0d289b917b70e5e95a3684920b51ae: Status 404 returned error can't find the container with id c5e7b75ddafef2c7bb94b9266bfed1d7ab0d289b917b70e5e95a3684920b51ae
Nov 03 12:32:37.174597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.174552    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"]
Nov 03 12:32:37.324318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.324245    4909 patch_prober.go:29] interesting pod/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 12:32:37.324318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.324290    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 12:32:37.785567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.785523    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" event={"ID":"c8c320c1-862e-4421-bd97-f3e64bd88d4d","Type":"ContainerStarted","Data":"09d96dfe5b93f84c284a6badc64cca67f7141ca5c9185f5eecb1ec5e5bb312af"}
Nov 03 12:32:37.785782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.785583    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" event={"ID":"c8c320c1-862e-4421-bd97-f3e64bd88d4d","Type":"ContainerStarted","Data":"7bc75e86df09f85a940e86fddecb3d83b797619bf02fc4fe6254172f6d09032b"}
Nov 03 12:32:37.785782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.785722    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:37.787158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.787129    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" event={"ID":"71ec8b30-32ec-407b-ad71-25dfdde68e79","Type":"ContainerStarted","Data":"fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274"}
Nov 03 12:32:37.787282 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.787174    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" event={"ID":"71ec8b30-32ec-407b-ad71-25dfdde68e79","Type":"ContainerStarted","Data":"c5e7b75ddafef2c7bb94b9266bfed1d7ab0d289b917b70e5e95a3684920b51ae"}
Nov 03 12:32:37.787282 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.787213    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:37.891944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.891890    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" podStartSLOduration=1.891876925 podStartE2EDuration="1.891876925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:37.889500942 +0000 UTC m=+2506.901967605" watchObservedRunningTime="2026-11-03 12:32:37.891876925 +0000 UTC m=+2506.904343588"
Nov 03 12:32:37.994113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:37.994054    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" podStartSLOduration=1.9940365230000001 podStartE2EDuration="1.994036523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:37.99233972 +0000 UTC m=+2507.004806382" watchObservedRunningTime="2026-11-03 12:32:37.994036523 +0000 UTC m=+2507.006503188"
Nov 03 12:32:38.324119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:38.324089    4909 patch_prober.go:29] interesting pod/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 12:32:38.324497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:38.324141    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 12:32:39.324038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:39.324002    4909 patch_prober.go:29] interesting pod/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 12:32:39.324210 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:39.324058    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 12:32:39.591901 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:32:39.591816    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:32:39.697540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:39.697507    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 12:32:40.328074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:40.328043    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:32:40.328457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:40.328426    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:32:40.329056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:40.329039    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:32:40.707531 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:40.707506    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 12:32:46.756922 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:46.756888    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"]
Nov 03 12:32:46.757396 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:46.757175    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy" containerID="cri-o://e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a" gracePeriod=30
Nov 03 12:32:47.231599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:47.231552    4909 patch_prober.go:29] interesting pod/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:32:47.231778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:47.231645    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:32:48.797793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:48.797765    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 12:32:48.798205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:48.797841    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 12:32:49.571875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:49.571832    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:32:52.519523 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.519494    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:32:52.653437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653354    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-kube-api-access-gw4q6\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-kube-api-access-gw4q6\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653395    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-socket\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653464    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-credential-socket\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653487    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-envoy\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653569    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b72109-dd79-4824-a3de-1c1b87f520f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b72109-dd79-4824-a3de-1c1b87f520f8-istiod-ca-cert\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653604    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-certs\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653664    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-data\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653945 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653726    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-podinfo\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653945 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653796    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-token\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-token\") pod \"04b72109-dd79-4824-a3de-1c1b87f520f8\" (UID: \"04b72109-dd79-4824-a3de-1c1b87f520f8\") "
Nov 03 12:32:52.653945 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653806    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-credential-socket" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:52.653945 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.653822    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-socket" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:52.654162 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.654005    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-data" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:52.654162 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.654020    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-certs" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:52.654484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.654367    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.654484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.654397    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.654484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.654414    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.654484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.654429    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.654484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.654367    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b72109-dd79-4824-a3de-1c1b87f520f8-istiod-ca-cert" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:52.655802 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.655777    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-kube-api-access-gw4q6" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "kube-api-access-gw4q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:52.656004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.655986    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-podinfo" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:32:52.656224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.656209    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-token" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:52.656290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.656269    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-envoy" pod "04b72109-dd79-4824-a3de-1c1b87f520f8" (UID: "04b72109-dd79-4824-a3de-1c1b87f520f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:52.755757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.755716    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b72109-dd79-4824-a3de-1c1b87f520f8-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.755757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.755748    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.755757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.755758    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.756036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.755769    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4q6\" (UniqueName: \"kubernetes.io/projected/04b72109-dd79-4824-a3de-1c1b87f520f8-kube-api-access-gw4q6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.756036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.755778    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b72109-dd79-4824-a3de-1c1b87f520f8-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:52.987117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.987085    4909 generic.go:356] "Generic (PLEG): container finished" podID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerID="e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a" exitCode=0
Nov 03 12:32:52.987368 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.987133    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"
Nov 03 12:32:52.987479 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.987152    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" event={"ID":"04b72109-dd79-4824-a3de-1c1b87f520f8","Type":"ContainerDied","Data":"e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a"}
Nov 03 12:32:52.987540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.987489    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v" event={"ID":"04b72109-dd79-4824-a3de-1c1b87f520f8","Type":"ContainerDied","Data":"412332a0c111bb3794a5a1b81c292869020a0d5e9caa97a9a33a83aa39e7bbc7"}
Nov 03 12:32:52.987540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.987518    4909 scope.go:122] "RemoveContainer" containerID="e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a"
Nov 03 12:32:52.998411 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:52.998393    4909 scope.go:122] "RemoveContainer" containerID="18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4"
Nov 03 12:32:53.009855 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.009828    4909 scope.go:122] "RemoveContainer" containerID="e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a"
Nov 03 12:32:53.010405 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:32:53.010124    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a\": container with ID starting with e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a not found: ID does not exist" containerID="e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a"
Nov 03 12:32:53.010405 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.010161    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a"} err="failed to get container status \"e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a\": rpc error: code = NotFound desc = could not find container \"e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a\": container with ID starting with e39914675e1351280ca57abb3e51e1f3aa8a88bbf77cc1c6abe7d9c10e52a99a not found: ID does not exist"
Nov 03 12:32:53.010405 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.010190    4909 scope.go:122] "RemoveContainer" containerID="18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4"
Nov 03 12:32:53.010563 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:32:53.010462    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4\": container with ID starting with 18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4 not found: ID does not exist" containerID="18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4"
Nov 03 12:32:53.010563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.010484    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4"} err="failed to get container status \"18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4\": rpc error: code = NotFound desc = could not find container \"18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4\": container with ID starting with 18482e33315028db18a9d93741430cb9033709285800833ad8a019c88da7b9a4 not found: ID does not exist"
Nov 03 12:32:53.143965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.143931    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"]
Nov 03 12:32:53.147392 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.147368    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2daac7-openshift-default-54576558f7-27w5v"]
Nov 03 12:32:53.177495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.177469    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"]
Nov 03 12:32:53.177980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.177967    4909 cpu_manager.go:414] "containerMap: removing container" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy"
Nov 03 12:32:53.178026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.177981    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy"
Nov 03 12:32:53.178097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.178088    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy"
Nov 03 12:32:53.182896 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.182882    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.231533 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.231512    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cacb84\""
Nov 03 12:32:53.231823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.231802    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cacb84-dockercfg-rzj8v\""
Nov 03 12:32:53.247952 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.247892    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"]
Nov 03 12:32:53.287735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.284760    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdr7\" (UniqueName: \"kubernetes.io/projected/cc822e2b-def1-4eb9-a22b-f03dc395205c-kube-api-access-4sdr7\") pod \"payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.287735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.284888    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc822e2b-def1-4eb9-a22b-f03dc395205c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.387545 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.387509    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"]
Nov 03 12:32:53.388679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.388648    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdr7\" (UniqueName: \"kubernetes.io/projected/cc822e2b-def1-4eb9-a22b-f03dc395205c-kube-api-access-4sdr7\") pod \"payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.389012 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.388989    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc822e2b-def1-4eb9-a22b-f03dc395205c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.389100 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.389019    4909 cpu_manager.go:414] "containerMap: removing container" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy"
Nov 03 12:32:53.389100 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.389038    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy"
Nov 03 12:32:53.389219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.389164    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" containerName="istio-proxy"
Nov 03 12:32:53.390038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.389995    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc822e2b-def1-4eb9-a22b-f03dc395205c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.394130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.394111    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.405585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.405558    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"]
Nov 03 12:32:53.444311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.444286    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdr7\" (UniqueName: \"kubernetes.io/projected/cc822e2b-def1-4eb9-a22b-f03dc395205c-kube-api-access-4sdr7\") pod \"payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.490899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.490869    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mv56\" (UniqueName: \"kubernetes.io/projected/4958a745-d677-4897-8d31-cf29f2a67b9c-kube-api-access-5mv56\") pod \"payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.491041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.491009    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4958a745-d677-4897-8d31-cf29f2a67b9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.496666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.496636    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:53.592039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.591994    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mv56\" (UniqueName: \"kubernetes.io/projected/4958a745-d677-4897-8d31-cf29f2a67b9c-kube-api-access-5mv56\") pod \"payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.592476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.592175    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4958a745-d677-4897-8d31-cf29f2a67b9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.593174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.593147    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4958a745-d677-4897-8d31-cf29f2a67b9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.645319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.645296    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mv56\" (UniqueName: \"kubernetes.io/projected/4958a745-d677-4897-8d31-cf29f2a67b9c-kube-api-access-5mv56\") pod \"payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.714802 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.714775    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:53.866445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.866421    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"]
Nov 03 12:32:53.868279 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:53.868249    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc822e2b_def1_4eb9_a22b_f03dc395205c.slice/crio-c3929039df1e7ba254a6adc8f2f99ce95718548cd1c4a20a6022aa4e9a5ee0c4 WatchSource:0}: Error finding container c3929039df1e7ba254a6adc8f2f99ce95718548cd1c4a20a6022aa4e9a5ee0c4: Status 404 returned error can't find the container with id c3929039df1e7ba254a6adc8f2f99ce95718548cd1c4a20a6022aa4e9a5ee0c4
Nov 03 12:32:53.995191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.995162    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" event={"ID":"cc822e2b-def1-4eb9-a22b-f03dc395205c","Type":"ContainerStarted","Data":"e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2"}
Nov 03 12:32:53.995305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.995210    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" event={"ID":"cc822e2b-def1-4eb9-a22b-f03dc395205c","Type":"ContainerStarted","Data":"c3929039df1e7ba254a6adc8f2f99ce95718548cd1c4a20a6022aa4e9a5ee0c4"}
Nov 03 12:32:53.995305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:53.995282    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:32:54.084016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:54.083992    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"]
Nov 03 12:32:54.085857 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:54.085815    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4958a745_d677_4897_8d31_cf29f2a67b9c.slice/crio-0efb6cca3bc289d05545f58baf7615648bf40d49655db2336febee2f1a8da5f2 WatchSource:0}: Error finding container 0efb6cca3bc289d05545f58baf7615648bf40d49655db2336febee2f1a8da5f2: Status 404 returned error can't find the container with id 0efb6cca3bc289d05545f58baf7615648bf40d49655db2336febee2f1a8da5f2
Nov 03 12:32:54.103869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:54.103828    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" podStartSLOduration=1.103814747 podStartE2EDuration="1.103814747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:54.10087902 +0000 UTC m=+2523.113345695" watchObservedRunningTime="2026-11-03 12:32:54.103814747 +0000 UTC m=+2523.116281409"
Nov 03 12:32:55.002776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:55.002734    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" event={"ID":"4958a745-d677-4897-8d31-cf29f2a67b9c","Type":"ContainerStarted","Data":"39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080"}
Nov 03 12:32:55.003174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:55.002801    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" event={"ID":"4958a745-d677-4897-8d31-cf29f2a67b9c","Type":"ContainerStarted","Data":"0efb6cca3bc289d05545f58baf7615648bf40d49655db2336febee2f1a8da5f2"}
Nov 03 12:32:55.003174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:55.003098    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:32:55.108446 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:55.108395    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" podStartSLOduration=2.108383154 podStartE2EDuration="2.108383154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:55.107817823 +0000 UTC m=+2524.120284486" watchObservedRunningTime="2026-11-03 12:32:55.108383154 +0000 UTC m=+2524.120849816"
Nov 03 12:32:55.125655 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:55.125624    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b72109-dd79-4824-a3de-1c1b87f520f8" path="/var/lib/kubelet/pods/04b72109-dd79-4824-a3de-1c1b87f520f8/volumes"
Nov 03 12:32:57.742426 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.742393    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"]
Nov 03 12:32:57.747546 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.747523    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:57.779110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.779087    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844132c2-044e-402d-9e2a-92c829b6f0bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:57.779204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.779120    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwk5h\" (UniqueName: \"kubernetes.io/projected/844132c2-044e-402d-9e2a-92c829b6f0bd-kube-api-access-bwk5h\") pod \"payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:57.916832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.916788    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844132c2-044e-402d-9e2a-92c829b6f0bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:57.917117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.917094    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwk5h\" (UniqueName: \"kubernetes.io/projected/844132c2-044e-402d-9e2a-92c829b6f0bd-kube-api-access-bwk5h\") pod \"payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:57.920208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.918838    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"]
Nov 03 12:32:57.920850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.920822    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bbca30-dockercfg-xzpzf\""
Nov 03 12:32:57.920980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.920885    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bbca30\""
Nov 03 12:32:57.925221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.925193    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"]
Nov 03 12:32:57.930183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.930121    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844132c2-044e-402d-9e2a-92c829b6f0bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:57.930531 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.930514    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:57.970174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.970155    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwk5h\" (UniqueName: \"kubernetes.io/projected/844132c2-044e-402d-9e2a-92c829b6f0bd-kube-api-access-bwk5h\") pod \"payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:57.985449 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:57.985428    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"]
Nov 03 12:32:58.018879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.018818    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7cv\" (UniqueName: \"kubernetes.io/projected/24244784-15a7-442c-a408-b132052c07df-kube-api-access-ph7cv\") pod \"payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:58.018989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.018960    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24244784-15a7-442c-a408-b132052c07df-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:58.062042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.062016    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:58.120528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.120497    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24244784-15a7-442c-a408-b132052c07df-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:58.120776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.120611    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7cv\" (UniqueName: \"kubernetes.io/projected/24244784-15a7-442c-a408-b132052c07df-kube-api-access-ph7cv\") pod \"payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:58.122503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.122461    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24244784-15a7-442c-a408-b132052c07df-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:58.174228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.174199    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7cv\" (UniqueName: \"kubernetes.io/projected/24244784-15a7-442c-a408-b132052c07df-kube-api-access-ph7cv\") pod \"payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:58.246144 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.246117    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:58.433098 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.433073    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"]
Nov 03 12:32:58.434430 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:58.434403    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844132c2_044e_402d_9e2a_92c829b6f0bd.slice/crio-2a084cfed5dfcc40de2b28234f18f0f619bc257bfb42507b365721a8d075313c WatchSource:0}: Error finding container 2a084cfed5dfcc40de2b28234f18f0f619bc257bfb42507b365721a8d075313c: Status 404 returned error can't find the container with id 2a084cfed5dfcc40de2b28234f18f0f619bc257bfb42507b365721a8d075313c
Nov 03 12:32:58.610631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:58.610606    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"]
Nov 03 12:32:58.612070 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:32:58.612043    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24244784_15a7_442c_a408_b132052c07df.slice/crio-5f1df2703cb790a0fa02fc1a7f20ddf1a46fb63a6974aa6806859ee79504e33d WatchSource:0}: Error finding container 5f1df2703cb790a0fa02fc1a7f20ddf1a46fb63a6974aa6806859ee79504e33d: Status 404 returned error can't find the container with id 5f1df2703cb790a0fa02fc1a7f20ddf1a46fb63a6974aa6806859ee79504e33d
Nov 03 12:32:59.028320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.028283    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp" event={"ID":"24244784-15a7-442c-a408-b132052c07df","Type":"ContainerStarted","Data":"c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375"}
Nov 03 12:32:59.028851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.028350    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp" event={"ID":"24244784-15a7-442c-a408-b132052c07df","Type":"ContainerStarted","Data":"5f1df2703cb790a0fa02fc1a7f20ddf1a46fb63a6974aa6806859ee79504e33d"}
Nov 03 12:32:59.028851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.028383    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:32:59.029861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.029835    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss" event={"ID":"844132c2-044e-402d-9e2a-92c829b6f0bd","Type":"ContainerStarted","Data":"93a10ccbbc25416df531aeccae7d221f6c196c887059aa148d3ac94018f9e4d7"}
Nov 03 12:32:59.029966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.029886    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss" event={"ID":"844132c2-044e-402d-9e2a-92c829b6f0bd","Type":"ContainerStarted","Data":"2a084cfed5dfcc40de2b28234f18f0f619bc257bfb42507b365721a8d075313c"}
Nov 03 12:32:59.029966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.029922    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:32:59.133322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.133278    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp" podStartSLOduration=2.133266499 podStartE2EDuration="2.133266499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:59.132605956 +0000 UTC m=+2528.145072618" watchObservedRunningTime="2026-11-03 12:32:59.133266499 +0000 UTC m=+2528.145733161"
Nov 03 12:32:59.236099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:32:59.236051    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss" podStartSLOduration=2.236038529 podStartE2EDuration="2.236038529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:59.23556307 +0000 UTC m=+2528.248029740" watchObservedRunningTime="2026-11-03 12:32:59.236038529 +0000 UTC m=+2528.248505186"
Nov 03 12:33:00.023154 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.023120    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"]
Nov 03 12:33:00.027473 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.027452    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.044388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.044362    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pf6l\" (UniqueName: \"kubernetes.io/projected/916f7473-2148-476c-a59c-1ec89776f020-kube-api-access-2pf6l\") pod \"payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.044683 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.044406    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f7473-2148-476c-a59c-1ec89776f020-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.103180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.103140    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"]
Nov 03 12:33:00.104285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.104253    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a4455f-dockercfg-hvxw6\""
Nov 03 12:33:00.131724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.127937    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a4455f\""
Nov 03 12:33:00.146726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.146667    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pf6l\" (UniqueName: \"kubernetes.io/projected/916f7473-2148-476c-a59c-1ec89776f020-kube-api-access-2pf6l\") pod \"payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.147193 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.147163    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f7473-2148-476c-a59c-1ec89776f020-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.161759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.150989    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f7473-2148-476c-a59c-1ec89776f020-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.187913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.187882    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"]
Nov 03 12:33:00.192687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.192658    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.207775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.207521    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pf6l\" (UniqueName: \"kubernetes.io/projected/916f7473-2148-476c-a59c-1ec89776f020-kube-api-access-2pf6l\") pod \"payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.248278 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.247652    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"]
Nov 03 12:33:00.249621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.249584    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8cd76c-1938-4645-bd13-ac37ca22f213-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.249872 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.249821    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fm46\" (UniqueName: \"kubernetes.io/projected/5e8cd76c-1938-4645-bd13-ac37ca22f213-kube-api-access-6fm46\") pod \"payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.343567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.343478    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:00.351204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.351172    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fm46\" (UniqueName: \"kubernetes.io/projected/5e8cd76c-1938-4645-bd13-ac37ca22f213-kube-api-access-6fm46\") pod \"payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.351345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.351254    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8cd76c-1938-4645-bd13-ac37ca22f213-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.352061 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.352032    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8cd76c-1938-4645-bd13-ac37ca22f213-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.405981 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.405589    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fm46\" (UniqueName: \"kubernetes.io/projected/5e8cd76c-1938-4645-bd13-ac37ca22f213-kube-api-access-6fm46\") pod \"payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.513172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.513139    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:00.709930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.709906    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"]
Nov 03 12:33:00.711376 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:33:00.711343    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916f7473_2148_476c_a59c_1ec89776f020.slice/crio-eb335f9add0618b624f4576599db5a667a53cb70fcb3937ea4c25163cf032134 WatchSource:0}: Error finding container eb335f9add0618b624f4576599db5a667a53cb70fcb3937ea4c25163cf032134: Status 404 returned error can't find the container with id eb335f9add0618b624f4576599db5a667a53cb70fcb3937ea4c25163cf032134
Nov 03 12:33:00.883253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:00.883226    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"]
Nov 03 12:33:00.884533 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:33:00.884504    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8cd76c_1938_4645_bd13_ac37ca22f213.slice/crio-3714080a841e31873bed4b203311125d3b96307443c3d1230ef735082d7eb614 WatchSource:0}: Error finding container 3714080a841e31873bed4b203311125d3b96307443c3d1230ef735082d7eb614: Status 404 returned error can't find the container with id 3714080a841e31873bed4b203311125d3b96307443c3d1230ef735082d7eb614
Nov 03 12:33:01.044594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.044518    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" event={"ID":"916f7473-2148-476c-a59c-1ec89776f020","Type":"ContainerStarted","Data":"d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383"}
Nov 03 12:33:01.044594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.044574    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" event={"ID":"916f7473-2148-476c-a59c-1ec89776f020","Type":"ContainerStarted","Data":"eb335f9add0618b624f4576599db5a667a53cb70fcb3937ea4c25163cf032134"}
Nov 03 12:33:01.045071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.044603    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:01.046242 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.046215    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" event={"ID":"5e8cd76c-1938-4645-bd13-ac37ca22f213","Type":"ContainerStarted","Data":"83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c"}
Nov 03 12:33:01.046356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.046255    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" event={"ID":"5e8cd76c-1938-4645-bd13-ac37ca22f213","Type":"ContainerStarted","Data":"3714080a841e31873bed4b203311125d3b96307443c3d1230ef735082d7eb614"}
Nov 03 12:33:01.046406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.046352    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:01.151351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.151289    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" podStartSLOduration=2.151270706 podStartE2EDuration="2.151270706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:01.148827232 +0000 UTC m=+2530.161293892" watchObservedRunningTime="2026-11-03 12:33:01.151270706 +0000 UTC m=+2530.163737370"
Nov 03 12:33:01.256072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:01.256022    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" podStartSLOduration=1.256007582 podStartE2EDuration="1.256007582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:01.25464036 +0000 UTC m=+2530.267107035" watchObservedRunningTime="2026-11-03 12:33:01.256007582 +0000 UTC m=+2530.268474253"
Nov 03 12:33:05.006095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:05.006068    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 12:33:06.011135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:06.011109    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 12:33:09.564728 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:33:09.564671    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:33:10.041207 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:10.041177    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:33:10.041488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:10.041457    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:33:12.057842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:12.057806    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:33:12.058330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:12.057894    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:33:12.182849 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:12.182811    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:33:13.701767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.691568    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"]
Nov 03 12:33:13.729776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.729747    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.807322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.807275    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"]
Nov 03 12:33:13.928429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928386    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.928618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928447    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.928618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928494    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.928618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928539    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqsw\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-kube-api-access-whqsw\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.928618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928584    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.928859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928632    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.928859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928775    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.928859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928804    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:13.929018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:13.928869    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.029779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.029667    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.029948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.029817    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.029948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.029859    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.029948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.029902    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.029948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.029943    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqsw\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-kube-api-access-whqsw\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.029987    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.030036    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.030138    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.030167    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.030293    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030751 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.030672    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.030774    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.030886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.030783    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.031065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.031043    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.032798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.032767    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.034333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.034304    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.086058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.085995    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqsw\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-kube-api-access-whqsw\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.088153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.088126    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.362676 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.362592    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:14.792583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.792558    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"]
Nov 03 12:33:14.793608 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:33:14.793583    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaafbcb_e21f_4e13_8c6f_ecb83fb12a1d.slice/crio-e92c8f6141c2861f87dd93c181396b0d10d98cedfd9f1516a1009ac9c38c6bd0 WatchSource:0}: Error finding container e92c8f6141c2861f87dd93c181396b0d10d98cedfd9f1516a1009ac9c38c6bd0: Status 404 returned error can't find the container with id e92c8f6141c2861f87dd93c181396b0d10d98cedfd9f1516a1009ac9c38c6bd0
Nov 03 12:33:14.796090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.796053    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:33:14.796199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.796132    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:33:14.796199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:14.796179    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:33:15.198942 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:15.198909    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" event={"ID":"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d","Type":"ContainerStarted","Data":"3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091"}
Nov 03 12:33:15.199103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:15.198966    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" event={"ID":"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d","Type":"ContainerStarted","Data":"e92c8f6141c2861f87dd93c181396b0d10d98cedfd9f1516a1009ac9c38c6bd0"}
Nov 03 12:33:15.307191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:15.307133    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podStartSLOduration=2.307117157 podStartE2EDuration="2.307117157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:15.306279789 +0000 UTC m=+2544.318746453" watchObservedRunningTime="2026-11-03 12:33:15.307117157 +0000 UTC m=+2544.319583821"
Nov 03 12:33:15.362912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:15.362868    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:15.364831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:15.364800    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:33:15.364965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:15.364866    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:33:16.363185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:16.363080    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:33:16.363834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:16.363792    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:33:17.363129 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:17.363089    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:33:17.363319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:17.363160    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:33:18.363077 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:18.363035    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:33:18.363253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:18.363105    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:33:19.362993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:19.362953    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:33:19.363477 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:19.363011    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:33:20.363055 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:20.363020    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:33:20.363593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:20.363073    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:33:21.362891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:21.362860    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:33:21.363058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:21.362910    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:33:22.369652 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:22.369617    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:22.370387 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:22.370360    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:22.371260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:22.371235    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:33:30.608193 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.608153    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"]
Nov 03 12:33:30.613846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.613822    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.662287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.662231    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-cacb84-serving-cert\""
Nov 03 12:33:30.711028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.710998    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"]
Nov 03 12:33:30.739033 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.739003    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.739174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.739054    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.739219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.739174    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.739219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.739210    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvgb\" (UniqueName: \"kubernetes.io/projected/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-kube-api-access-7nvgb\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.854839 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.854783    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.864927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.859634    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvgb\" (UniqueName: \"kubernetes.io/projected/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-kube-api-access-7nvgb\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.864927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.860086    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.864927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.860278    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.873589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.855809    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.885988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.885952    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.886327 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.886298    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.923500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.923465    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvgb\" (UniqueName: \"kubernetes.io/projected/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-kube-api-access-7nvgb\") pod \"maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:30.932479 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:30.932451    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:31.298830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:31.298802    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"]
Nov 03 12:33:31.300424 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:33:31.300398    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4676bc_f7ae_4b2c_9d4b_39b10bb116d5.slice/crio-7839587e43fb65f4ffb54af240fb42741a37136bd36dfdb2994d3090805d1c9c WatchSource:0}: Error finding container 7839587e43fb65f4ffb54af240fb42741a37136bd36dfdb2994d3090805d1c9c: Status 404 returned error can't find the container with id 7839587e43fb65f4ffb54af240fb42741a37136bd36dfdb2994d3090805d1c9c
Nov 03 12:33:31.301838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:31.301819    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:33:32.303936 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:32.303903    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl" event={"ID":"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5","Type":"ContainerStarted","Data":"1a192d74ee32e7959dd9541d1c3fd02f042d36ea673526b0f50726e884203bbe"}
Nov 03 12:33:32.304305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:32.303954    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl" event={"ID":"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5","Type":"ContainerStarted","Data":"7839587e43fb65f4ffb54af240fb42741a37136bd36dfdb2994d3090805d1c9c"}
Nov 03 12:33:32.304305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:32.304098    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:32.413721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:32.413641    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl" podStartSLOduration=1.53576434 podStartE2EDuration="2.413624121s" totalImagesPullingTime="877.859781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:32.412911091 +0000 UTC m=+2561.425377805" watchObservedRunningTime="2026-11-03 12:33:32.413624121 +0000 UTC m=+2561.426090785"
Nov 03 12:33:37.376326 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:37.376294    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:33:38.316386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:38.316355    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 12:33:39.570060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.570027    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"]
Nov 03 12:33:39.575324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.575292    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.589498 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:33:39.589471    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:33:39.624371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.624345    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bbca30-serving-cert\""
Nov 03 12:33:39.673074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.673042    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"]
Nov 03 12:33:39.689963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.687253    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.689963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.687368    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.689963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.687412    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tw62\" (UniqueName: \"kubernetes.io/projected/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-kube-api-access-5tw62\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.689963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.687492    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-metrics-certs\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.788589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.788553    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-metrics-certs\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.788820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.788639    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.788820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.788728    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.788820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.788766    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tw62\" (UniqueName: \"kubernetes.io/projected/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-kube-api-access-5tw62\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.788820 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:33:39.788788    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-serving-cert: secret "maas-api-e2e-praxis-b-bbca30-serving-cert" not found
Nov 03 12:33:39.789069 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:33:39.788867    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls podName:3d41305d-7ab4-4011-ada5-72bdd4cb9d2c nodeName:}" failed. No retries permitted until 2026-11-03 12:33:40.288849391 +0000 UTC m=+2569.301316046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls") pod "maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w" (UID: "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c") : secret "maas-api-e2e-praxis-b-bbca30-serving-cert" not found
Nov 03 12:33:39.789397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.789379    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.791126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.791104    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-metrics-certs\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:39.871712 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:39.871612    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tw62\" (UniqueName: \"kubernetes.io/projected/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-kube-api-access-5tw62\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:40.303773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:40.303737    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:40.306197 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:40.306175    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:40.490085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:40.490034    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:40.883316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:40.883291    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"]
Nov 03 12:33:40.885568 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:33:40.885541    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d41305d_7ab4_4011_ada5_72bdd4cb9d2c.slice/crio-db830a220127d841e66683f351aa8e3dbd90759518fc5deb2da0dd285e6ababa WatchSource:0}: Error finding container db830a220127d841e66683f351aa8e3dbd90759518fc5deb2da0dd285e6ababa: Status 404 returned error can't find the container with id db830a220127d841e66683f351aa8e3dbd90759518fc5deb2da0dd285e6ababa
Nov 03 12:33:41.383313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:41.383281    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w" event={"ID":"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c","Type":"ContainerStarted","Data":"db830a220127d841e66683f351aa8e3dbd90759518fc5deb2da0dd285e6ababa"}
Nov 03 12:33:42.392490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:42.392449    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w" event={"ID":"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c","Type":"ContainerStarted","Data":"e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4"}
Nov 03 12:33:42.392967 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:42.392666    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:42.508526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:42.508462    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w" podStartSLOduration=2.605332538 podStartE2EDuration="3.508444282s" totalImagesPullingTime="903.111744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:42.504303202 +0000 UTC m=+2571.516769881" watchObservedRunningTime="2026-11-03 12:33:42.508444282 +0000 UTC m=+2571.520910947"
Nov 03 12:33:44.164452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.164415    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"]
Nov 03 12:33:44.169904 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.169879    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.210454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.210429    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-metrics-certs\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.210568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.210478    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv5v\" (UniqueName: \"kubernetes.io/projected/70caac4b-48cc-48af-b03e-5d5c4e943421-kube-api-access-xnv5v\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.210765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.210743    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.210816 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.210799    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70caac4b-48cc-48af-b03e-5d5c4e943421-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.220847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.220825    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"]
Nov 03 12:33:44.262849 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.262825    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a4455f-serving-cert\""
Nov 03 12:33:44.312089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.312051    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-metrics-certs\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.312243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.312120    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv5v\" (UniqueName: \"kubernetes.io/projected/70caac4b-48cc-48af-b03e-5d5c4e943421-kube-api-access-xnv5v\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.312317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.312257    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.312317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.312304    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70caac4b-48cc-48af-b03e-5d5c4e943421-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.312472 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:33:44.312451    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-serving-cert: secret "maas-api-e2e-praxis-a-a4455f-serving-cert" not found
Nov 03 12:33:44.312535 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:33:44.312519    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls podName:70caac4b-48cc-48af-b03e-5d5c4e943421 nodeName:}" failed. No retries permitted until 2026-11-03 12:33:44.812499356 +0000 UTC m=+2573.824966001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls") pod "maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk" (UID: "70caac4b-48cc-48af-b03e-5d5c4e943421") : secret "maas-api-e2e-praxis-a-a4455f-serving-cert" not found
Nov 03 12:33:44.313024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.313002    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70caac4b-48cc-48af-b03e-5d5c4e943421-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.315311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.315287    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-metrics-certs\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.365009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.364982    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv5v\" (UniqueName: \"kubernetes.io/projected/70caac4b-48cc-48af-b03e-5d5c4e943421-kube-api-access-xnv5v\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.857094 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.857046    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:44.859863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:44.859831    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:45.086103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:45.086058    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:45.462273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:45.462232    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"]
Nov 03 12:33:45.466745 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:33:45.466711    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70caac4b_48cc_48af_b03e_5d5c4e943421.slice/crio-467b0ada4069571f329ae5806f19f2e7428a43d49118576912ef7fbd5e194c54 WatchSource:0}: Error finding container 467b0ada4069571f329ae5806f19f2e7428a43d49118576912ef7fbd5e194c54: Status 404 returned error can't find the container with id 467b0ada4069571f329ae5806f19f2e7428a43d49118576912ef7fbd5e194c54
Nov 03 12:33:46.483871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:46.483830    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk" event={"ID":"70caac4b-48cc-48af-b03e-5d5c4e943421","Type":"ContainerStarted","Data":"bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e"}
Nov 03 12:33:46.485566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:46.485541    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:46.485990 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:46.485951    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk" event={"ID":"70caac4b-48cc-48af-b03e-5d5c4e943421","Type":"ContainerStarted","Data":"467b0ada4069571f329ae5806f19f2e7428a43d49118576912ef7fbd5e194c54"}
Nov 03 12:33:46.603641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:46.603539    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk" podStartSLOduration=1.735324962 podStartE2EDuration="2.603525852s" totalImagesPullingTime="868.20089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:46.603260003 +0000 UTC m=+2575.615726678" watchObservedRunningTime="2026-11-03 12:33:46.603525852 +0000 UTC m=+2575.615992515"
Nov 03 12:33:46.749829 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:46.748720    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:33:46.749829 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:46.748796    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:33:47.492494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:47.492436    4909 generic.go:356] "Generic (PLEG): container finished" podID="396f2942-0950-46ad-812b-d506efc6bfbd" containerID="bf75a593aa5f87ca9835edeb6baad3311b9567c931adab7b63caf4512bd5469d" exitCode=0
Nov 03 12:33:47.492928 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:47.492505    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" event={"ID":"396f2942-0950-46ad-812b-d506efc6bfbd","Type":"ContainerDied","Data":"bf75a593aa5f87ca9835edeb6baad3311b9567c931adab7b63caf4512bd5469d"}
Nov 03 12:33:47.493083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:47.493066    4909 scope.go:122] "RemoveContainer" containerID="bf75a593aa5f87ca9835edeb6baad3311b9567c931adab7b63caf4512bd5469d"
Nov 03 12:33:47.494371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:47.494342    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:33:47.494456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:47.494410    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:33:47.494456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:47.494440    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:33:48.404036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:48.404010    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:33:48.501772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:48.501732    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" event={"ID":"396f2942-0950-46ad-812b-d506efc6bfbd","Type":"ContainerStarted","Data":"87412adf4f4c7f39820e8237768b3fbc7e219e016b0f068d231524cc33ad0f5e"}
Nov 03 12:33:49.450029 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:49.449997    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:33:49.451900 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:49.450799    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 12:33:49.452103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:49.452075    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 12:33:50.458719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:50.451040    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 12:33:50.458719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:50.451137    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 12:33:51.455721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:51.448997    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 12:33:51.455721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:51.449063    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 12:33:52.455718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:52.450992    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 12:33:52.455718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:52.451050    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 12:33:53.454721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:53.450072    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 12:33:53.454721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:53.450134    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 12:33:53.508203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:53.508169    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:33:54.451669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:54.451639    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:33:54.452121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:54.452096    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:33:54.452757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:33:54.452740    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 12:34:03.713172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.713139    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"]
Nov 03 12:34:03.719186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.719165    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.767233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.767211    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bbca30-openshift-default-dockercfg-xx7jx\""
Nov 03 12:34:03.795132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795104    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795231 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795155    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795245    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36527505-c91a-4919-af7c-24a516e2fc81-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795373 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795359    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795397    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36527505-c91a-4919-af7c-24a516e2fc81-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795429    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795514    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895wc\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-kube-api-access-895wc\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795548    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.795694 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.795675    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.816715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.816677    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"]
Nov 03 12:34:03.896866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.896837    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.896880    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36527505-c91a-4919-af7c-24a516e2fc81-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.896913    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.896950    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895wc\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-kube-api-access-895wc\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.896981    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897177 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897072    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897177 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897103    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897177 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897140    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897177 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897166    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36527505-c91a-4919-af7c-24a516e2fc81-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897394    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897543 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897518    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897627 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897538    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897639    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.897772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.897684    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36527505-c91a-4919-af7c-24a516e2fc81-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.899239 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.899219    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.899503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.899485    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36527505-c91a-4919-af7c-24a516e2fc81-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.949494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.949467    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:03.949923 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:03.949897    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895wc\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-kube-api-access-895wc\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:04.032394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.032321    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:04.452846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.452787    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"]
Nov 03 12:34:04.453654 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:34:04.453623    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36527505_c91a_4919_af7c_24a516e2fc81.slice/crio-42692fb225182aed288d7c350ed5e0b20f4f431c81e9998782f8712d73db0a3f WatchSource:0}: Error finding container 42692fb225182aed288d7c350ed5e0b20f4f431c81e9998782f8712d73db0a3f: Status 404 returned error can't find the container with id 42692fb225182aed288d7c350ed5e0b20f4f431c81e9998782f8712d73db0a3f
Nov 03 12:34:04.456170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.456134    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:04.456275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.456205    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:04.456275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.456235    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:04.612907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.612874    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" event={"ID":"36527505-c91a-4919-af7c-24a516e2fc81","Type":"ContainerStarted","Data":"ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03"}
Nov 03 12:34:04.613107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.612929    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" event={"ID":"36527505-c91a-4919-af7c-24a516e2fc81","Type":"ContainerStarted","Data":"42692fb225182aed288d7c350ed5e0b20f4f431c81e9998782f8712d73db0a3f"}
Nov 03 12:34:04.721531 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:04.721430    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podStartSLOduration=1.721414249 podStartE2EDuration="1.721414249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:04.719964016 +0000 UTC m=+2593.732430678" watchObservedRunningTime="2026-11-03 12:34:04.721414249 +0000 UTC m=+2593.733880913"
Nov 03 12:34:05.033015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:05.032936    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:05.034854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:05.034829    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:34:05.034971 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:05.034877    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 12:34:06.032652 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:06.032614    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:34:06.033103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:06.032670    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 12:34:07.033034 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:07.033000    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:34:07.033434 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:07.033055    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 12:34:08.033623 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:08.033586    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:34:08.034032 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:08.033641    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 12:34:09.032890 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:09.032858    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:34:09.033063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:09.032915    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 12:34:09.585901 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:34:09.585868    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:34:10.032974 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:10.032938    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:34:10.033157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:10.032993    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 12:34:11.069204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:11.069170    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:11.075130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:11.075101    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:11.076600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:11.076573    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:34:21.649638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:21.649600    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:34:24.122130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.122094    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"]
Nov 03 12:34:24.126600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.126576    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.177690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.177661    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"]
Nov 03 12:34:24.197747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.197689    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f9d76a-293c-4eed-bca2-8bbaae21de1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.197924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.197789    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzbh\" (UniqueName: \"kubernetes.io/projected/92f9d76a-293c-4eed-bca2-8bbaae21de1b-kube-api-access-6pzbh\") pod \"payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.219931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.219903    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-743d9b\""
Nov 03 12:34:24.220040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.220006    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-743d9b-dockercfg-4729h\""
Nov 03 12:34:24.299030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.299002    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzbh\" (UniqueName: \"kubernetes.io/projected/92f9d76a-293c-4eed-bca2-8bbaae21de1b-kube-api-access-6pzbh\") pod \"payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.299178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.299165    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f9d76a-293c-4eed-bca2-8bbaae21de1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.299858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.299838    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f9d76a-293c-4eed-bca2-8bbaae21de1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.351239 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.351216    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzbh\" (UniqueName: \"kubernetes.io/projected/92f9d76a-293c-4eed-bca2-8bbaae21de1b-kube-api-access-6pzbh\") pod \"payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.441350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.441328    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:24.809760 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.809737    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"]
Nov 03 12:34:24.811826 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:34:24.811796    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f9d76a_293c_4eed_bca2_8bbaae21de1b.slice/crio-d5bae86649923d8dc67cb651fdc641b44b0b472bf71ed30341c83276ca196db1 WatchSource:0}: Error finding container d5bae86649923d8dc67cb651fdc641b44b0b472bf71ed30341c83276ca196db1: Status 404 returned error can't find the container with id d5bae86649923d8dc67cb651fdc641b44b0b472bf71ed30341c83276ca196db1
Nov 03 12:34:24.842736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:24.842694    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" event={"ID":"92f9d76a-293c-4eed-bca2-8bbaae21de1b","Type":"ContainerStarted","Data":"d5bae86649923d8dc67cb651fdc641b44b0b472bf71ed30341c83276ca196db1"}
Nov 03 12:34:25.850137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:25.850105    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" event={"ID":"92f9d76a-293c-4eed-bca2-8bbaae21de1b","Type":"ContainerStarted","Data":"5f8fac862e9bf0009b9f0932fa4631ca65a4e9ee3178af4c9ee13f11c4a91a3f"}
Nov 03 12:34:25.850552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:25.850173    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:25.956784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:25.956739    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" podStartSLOduration=1.956724508 podStartE2EDuration="1.956724508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:25.954011603 +0000 UTC m=+2614.966478265" watchObservedRunningTime="2026-11-03 12:34:25.956724508 +0000 UTC m=+2614.969191169"
Nov 03 12:34:36.859787 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:36.859755    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 12:34:37.696793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.696757    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"]
Nov 03 12:34:37.702072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.702054    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.750743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.750193    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e0615c-openshift-default-dockercfg-pjmgq\""
Nov 03 12:34:37.782105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782076    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkwm\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-kube-api-access-4fkwm\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782204    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-socket\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782248    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-podinfo\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782386 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782364    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-data\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782413    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-envoy\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782534    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-token\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782581    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8698500-d104-43a2-bb62-aae06cc3a4df-istiod-ca-cert\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782613    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-certs\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.782773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.782752    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-credential-socket\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.799659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.799632    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"]
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.884778    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-credential-socket\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.884848    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkwm\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-kube-api-access-4fkwm\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.884979    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-socket\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.885016    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-podinfo\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.885100    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-data\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.885155    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-envoy\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.885240    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-token\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.885301    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8698500-d104-43a2-bb62-aae06cc3a4df-istiod-ca-cert\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.885330    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-certs\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.885868    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-certs\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.886133    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-credential-socket\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.890720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.886746    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-socket\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.931720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.928537    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8698500-d104-43a2-bb62-aae06cc3a4df-istiod-ca-cert\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.965824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.965743    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-data\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.980114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.980080    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-envoy\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.980356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.980325    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-podinfo\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:37.994731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:37.994681    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkwm\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-kube-api-access-4fkwm\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:38.015727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.015431    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-token\") pod \"e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") " pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:38.016164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.015950    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:38.477438 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:34:38.477409    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8698500_d104_43a2_bb62_aae06cc3a4df.slice/crio-5c36b83d5e2a4c787d69c9151c03d5eec61be03eb98a441f245d5dc8d767209f WatchSource:0}: Error finding container 5c36b83d5e2a4c787d69c9151c03d5eec61be03eb98a441f245d5dc8d767209f: Status 404 returned error can't find the container with id 5c36b83d5e2a4c787d69c9151c03d5eec61be03eb98a441f245d5dc8d767209f
Nov 03 12:34:38.477568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.477515    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"]
Nov 03 12:34:38.480132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.480096    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:38.480225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.480175    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:38.480225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.480213    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:38.951521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.951479    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" event={"ID":"f8698500-d104-43a2-bb62-aae06cc3a4df","Type":"ContainerStarted","Data":"b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd"}
Nov 03 12:34:38.951927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:38.951546    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" event={"ID":"f8698500-d104-43a2-bb62-aae06cc3a4df","Type":"ContainerStarted","Data":"5c36b83d5e2a4c787d69c9151c03d5eec61be03eb98a441f245d5dc8d767209f"}
Nov 03 12:34:39.018970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:39.018945    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:39.024649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:39.024629    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:39.060625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:39.060585    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" podStartSLOduration=2.060572669 podStartE2EDuration="2.060572669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:39.05840655 +0000 UTC m=+2628.070873244" watchObservedRunningTime="2026-11-03 12:34:39.060572669 +0000 UTC m=+2628.073039331"
Nov 03 12:34:39.565202 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:34:39.565168    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:34:39.958003 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:39.957968    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:39.959056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:39.959038    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:34:49.087105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.087070    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"]
Nov 03 12:34:49.091442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.091422    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.150828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.150801    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"]
Nov 03 12:34:49.162409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.162383    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.162559 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.162447    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.162641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.162586    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.162641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.162627    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.162805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.162741    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.162885 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.162867    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.163403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.163376    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.163584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.163565    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxfs\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-kube-api-access-bxxfs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.163741 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.163722    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265219    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265345    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265396    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265430    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxfs\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-kube-api-access-bxxfs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265463    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265560 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265519    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265617    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265658    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265660    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.265755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.265685    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.266022 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.266002    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.266535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.266506    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.266833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.266811    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.267328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.267306    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.268942 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.268918    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.269213 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.269195    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.318239 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.318211    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.318740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.318716    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxfs\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-kube-api-access-bxxfs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.405941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.405875    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:49.821525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.821492    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"]
Nov 03 12:34:49.822646 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:34:49.822619    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6361dd_8e6f_4669_acb7_e586f2dedde4.slice/crio-7c0d62fe85c9166af58fe5fc762088492882fa47dbc47d99caa40efaebb60c9b WatchSource:0}: Error finding container 7c0d62fe85c9166af58fe5fc762088492882fa47dbc47d99caa40efaebb60c9b: Status 404 returned error can't find the container with id 7c0d62fe85c9166af58fe5fc762088492882fa47dbc47d99caa40efaebb60c9b
Nov 03 12:34:49.825103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.825077    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:49.825173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.825137    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:49.825173 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:49.825163    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:34:50.020558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:50.020511    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" event={"ID":"5c6361dd-8e6f-4669-acb7-e586f2dedde4","Type":"ContainerStarted","Data":"765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747"}
Nov 03 12:34:50.020749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:50.020585    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" event={"ID":"5c6361dd-8e6f-4669-acb7-e586f2dedde4","Type":"ContainerStarted","Data":"7c0d62fe85c9166af58fe5fc762088492882fa47dbc47d99caa40efaebb60c9b"}
Nov 03 12:34:50.128965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:50.128866    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podStartSLOduration=1.12885307 podStartE2EDuration="1.12885307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:50.127165968 +0000 UTC m=+2639.139632647" watchObservedRunningTime="2026-11-03 12:34:50.12885307 +0000 UTC m=+2639.141319732"
Nov 03 12:34:50.407011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:50.406931    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:50.408710 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:50.408676    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 12:34:50.408846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:50.408747    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 12:34:51.407040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:51.406999    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 12:34:51.407455 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:51.407063    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 12:34:52.406459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:52.406426    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 12:34:52.406645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:52.406483    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 12:34:53.407190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:53.407143    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 12:34:53.407573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:53.407210    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 12:34:53.988987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:53.988946    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"]
Nov 03 12:34:54.089171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:54.089138    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"]
Nov 03 12:34:54.437805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:54.431088    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 12:34:54.437805 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:54.435540    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 12:34:55.414752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:55.412291    4909 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 12:34:55.414752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:55.412449    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 12:34:56.442183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:56.442151    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:56.442713 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:56.442675    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:56.443530 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:56.443509    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:34:58.990227 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:58.990189    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95" podUID="57674fbd-3ec2-4612-9abc-0145666d1fd2" containerName="payload-processing" containerID="cri-o://70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e" gracePeriod=25
Nov 03 12:34:59.089758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.089684    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2" podUID="fb8f3341-736d-4961-bb21-354243728c5b" containerName="payload-pre-processing" containerID="cri-o://9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca" gracePeriod=25
Nov 03 12:34:59.279408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.279306    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:34:59.384828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.384801    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57674fbd-3ec2-4612-9abc-0145666d1fd2-kube-api-access-vpf69\" (UniqueName: \"kubernetes.io/projected/57674fbd-3ec2-4612-9abc-0145666d1fd2-kube-api-access-vpf69\") pod \"57674fbd-3ec2-4612-9abc-0145666d1fd2\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") "
Nov 03 12:34:59.384970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.384953    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57674fbd-3ec2-4612-9abc-0145666d1fd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57674fbd-3ec2-4612-9abc-0145666d1fd2-plugins-config-volume\") pod \"57674fbd-3ec2-4612-9abc-0145666d1fd2\" (UID: \"57674fbd-3ec2-4612-9abc-0145666d1fd2\") "
Nov 03 12:34:59.385310 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.385285    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57674fbd-3ec2-4612-9abc-0145666d1fd2-plugins-config-volume" pod "57674fbd-3ec2-4612-9abc-0145666d1fd2" (UID: "57674fbd-3ec2-4612-9abc-0145666d1fd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:34:59.385620 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.385597    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57674fbd-3ec2-4612-9abc-0145666d1fd2-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:34:59.386980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.386957    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57674fbd-3ec2-4612-9abc-0145666d1fd2-kube-api-access-vpf69" pod "57674fbd-3ec2-4612-9abc-0145666d1fd2" (UID: "57674fbd-3ec2-4612-9abc-0145666d1fd2"). InnerVolumeSpecName "kube-api-access-vpf69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:34:59.390892 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.390874    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:34:59.447879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.447822    4909 generic.go:356] "Generic (PLEG): container finished" podID="fb8f3341-736d-4961-bb21-354243728c5b" containerID="9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca" exitCode=0
Nov 03 12:34:59.448047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.447887    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"
Nov 03 12:34:59.448047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.447888    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2" event={"ID":"fb8f3341-736d-4961-bb21-354243728c5b","Type":"ContainerDied","Data":"9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca"}
Nov 03 12:34:59.448047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.447967    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2" event={"ID":"fb8f3341-736d-4961-bb21-354243728c5b","Type":"ContainerDied","Data":"9d86b01bcf8a6c8fb2ef50b41dd922ffee73b1872a104f69e58da4a025b6123d"}
Nov 03 12:34:59.448047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.448022    4909 scope.go:122] "RemoveContainer" containerID="9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca"
Nov 03 12:34:59.450910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.450877    4909 generic.go:356] "Generic (PLEG): container finished" podID="57674fbd-3ec2-4612-9abc-0145666d1fd2" containerID="70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e" exitCode=0
Nov 03 12:34:59.451016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.450949    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95" event={"ID":"57674fbd-3ec2-4612-9abc-0145666d1fd2","Type":"ContainerDied","Data":"70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e"}
Nov 03 12:34:59.451016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.450994    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95" event={"ID":"57674fbd-3ec2-4612-9abc-0145666d1fd2","Type":"ContainerDied","Data":"9cac1c958bb0b6044bb005ebd0f661dd939256ee6041d9385e8a5935cfbbdfac"}
Nov 03 12:34:59.451137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.451029    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"
Nov 03 12:34:59.459570 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.459554    4909 scope.go:122] "RemoveContainer" containerID="9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca"
Nov 03 12:34:59.460144 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:34:59.459982    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca\": container with ID starting with 9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca not found: ID does not exist" containerID="9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca"
Nov 03 12:34:59.460144 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.460020    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca"} err="failed to get container status \"9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca\": rpc error: code = NotFound desc = could not find container \"9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca\": container with ID starting with 9be98119b34e6913e1a3731b5717f2f723f6586c0af91fa923b42eee58c58cca not found: ID does not exist"
Nov 03 12:34:59.460144 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.460046    4909 scope.go:122] "RemoveContainer" containerID="70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e"
Nov 03 12:34:59.468776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.468760    4909 scope.go:122] "RemoveContainer" containerID="70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e"
Nov 03 12:34:59.469016 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:34:59.469000    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e\": container with ID starting with 70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e not found: ID does not exist" containerID="70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e"
Nov 03 12:34:59.469069 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.469022    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e"} err="failed to get container status \"70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e\": rpc error: code = NotFound desc = could not find container \"70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e\": container with ID starting with 70bd3d926230b17df2855837ba9a51751cd46c7cc2504b052550d46036aa2f5e not found: ID does not exist"
Nov 03 12:34:59.486561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.486535    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8f3341-736d-4961-bb21-354243728c5b-kube-api-access-tsxg4\" (UniqueName: \"kubernetes.io/projected/fb8f3341-736d-4961-bb21-354243728c5b-kube-api-access-tsxg4\") pod \"fb8f3341-736d-4961-bb21-354243728c5b\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") "
Nov 03 12:34:59.486637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.486569    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8f3341-736d-4961-bb21-354243728c5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8f3341-736d-4961-bb21-354243728c5b-plugins-config-volume\") pod \"fb8f3341-736d-4961-bb21-354243728c5b\" (UID: \"fb8f3341-736d-4961-bb21-354243728c5b\") "
Nov 03 12:34:59.486962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.486940    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8f3341-736d-4961-bb21-354243728c5b-plugins-config-volume" pod "fb8f3341-736d-4961-bb21-354243728c5b" (UID: "fb8f3341-736d-4961-bb21-354243728c5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:34:59.487042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.486998    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpf69\" (UniqueName: \"kubernetes.io/projected/57674fbd-3ec2-4612-9abc-0145666d1fd2-kube-api-access-vpf69\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:34:59.488425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.488402    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8f3341-736d-4961-bb21-354243728c5b-kube-api-access-tsxg4" pod "fb8f3341-736d-4961-bb21-354243728c5b" (UID: "fb8f3341-736d-4961-bb21-354243728c5b"). InnerVolumeSpecName "kube-api-access-tsxg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:34:59.588761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.588678    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxg4\" (UniqueName: \"kubernetes.io/projected/fb8f3341-736d-4961-bb21-354243728c5b-kube-api-access-tsxg4\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:34:59.588761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.588744    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8f3341-736d-4961-bb21-354243728c5b-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:34:59.655296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.655241    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"]
Nov 03 12:34:59.659025 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.658999    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb4a0f-764b867747-ffb95"]
Nov 03 12:34:59.862957 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.862891    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"]
Nov 03 12:34:59.868072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:34:59.868046    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb4a0f-5fc99c754c-c6db2"]
Nov 03 12:35:01.174531 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.174496    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57674fbd-3ec2-4612-9abc-0145666d1fd2" path="/var/lib/kubelet/pods/57674fbd-3ec2-4612-9abc-0145666d1fd2/volumes"
Nov 03 12:35:01.175311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.175287    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8f3341-736d-4961-bb21-354243728c5b" path="/var/lib/kubelet/pods/fb8f3341-736d-4961-bb21-354243728c5b/volumes"
Nov 03 12:35:01.200287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.200255    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"]
Nov 03 12:35:01.202710 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.202666    4909 cpu_manager.go:414] "containerMap: removing container" podUID="57674fbd-3ec2-4612-9abc-0145666d1fd2" containerName="payload-processing"
Nov 03 12:35:01.202887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.202870    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57674fbd-3ec2-4612-9abc-0145666d1fd2" containerName="payload-processing"
Nov 03 12:35:01.203014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.203002    4909 cpu_manager.go:414] "containerMap: removing container" podUID="fb8f3341-736d-4961-bb21-354243728c5b" containerName="payload-pre-processing"
Nov 03 12:35:01.203091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.203079    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8f3341-736d-4961-bb21-354243728c5b" containerName="payload-pre-processing"
Nov 03 12:35:01.203563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.203544    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="57674fbd-3ec2-4612-9abc-0145666d1fd2" containerName="payload-processing"
Nov 03 12:35:01.203682 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.203671    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8f3341-736d-4961-bb21-354243728c5b" containerName="payload-pre-processing"
Nov 03 12:35:01.210518 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.210497    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:01.290198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.290004    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3176d9\""
Nov 03 12:35:01.306982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.306923    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3176d9-dockercfg-w4nm8\""
Nov 03 12:35:01.353895 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.350430    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m295t\" (UniqueName: \"kubernetes.io/projected/282ce652-1641-4658-a189-eeaff2904167-kube-api-access-m295t\") pod \"payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:01.383721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.358965    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282ce652-1641-4658-a189-eeaff2904167-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:01.386124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.354140    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"]
Nov 03 12:35:01.394570 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.394539    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"]
Nov 03 12:35:01.404242 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.404186    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.424738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.424643    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"]
Nov 03 12:35:01.486960 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.486930    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cjs\" (UniqueName: \"kubernetes.io/projected/11b73485-b056-4b49-991f-9be46681cba8-kube-api-access-j4cjs\") pod \"payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.487095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.486995    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m295t\" (UniqueName: \"kubernetes.io/projected/282ce652-1641-4658-a189-eeaff2904167-kube-api-access-m295t\") pod \"payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:01.487168 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.487148    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282ce652-1641-4658-a189-eeaff2904167-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:01.487208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.487183    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b73485-b056-4b49-991f-9be46681cba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.487845 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.487829    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282ce652-1641-4658-a189-eeaff2904167-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:01.539322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.539291    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m295t\" (UniqueName: \"kubernetes.io/projected/282ce652-1641-4658-a189-eeaff2904167-kube-api-access-m295t\") pod \"payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:01.588135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.588103    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b73485-b056-4b49-991f-9be46681cba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.588295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.588219    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cjs\" (UniqueName: \"kubernetes.io/projected/11b73485-b056-4b49-991f-9be46681cba8-kube-api-access-j4cjs\") pod \"payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.588780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.588761    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b73485-b056-4b49-991f-9be46681cba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.640745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.640720    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cjs\" (UniqueName: \"kubernetes.io/projected/11b73485-b056-4b49-991f-9be46681cba8-kube-api-access-j4cjs\") pod \"payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.723628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.723597    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:01.825174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:01.825136    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:02.091596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.091566    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"]
Nov 03 12:35:02.093283 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:35:02.093258    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b73485_b056_4b49_991f_9be46681cba8.slice/crio-016676204fd84b7a0a92d01c43db21c0e00f5710d15d2f296225b4ab95e21e54 WatchSource:0}: Error finding container 016676204fd84b7a0a92d01c43db21c0e00f5710d15d2f296225b4ab95e21e54: Status 404 returned error can't find the container with id 016676204fd84b7a0a92d01c43db21c0e00f5710d15d2f296225b4ab95e21e54
Nov 03 12:35:02.202081 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.202056    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"]
Nov 03 12:35:02.203436 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:35:02.203410    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282ce652_1641_4658_a189_eeaff2904167.slice/crio-aad5999b2d7394cb6d472a5eee688eedf160804ca8692bd5025e41a0f9a4fca3 WatchSource:0}: Error finding container aad5999b2d7394cb6d472a5eee688eedf160804ca8692bd5025e41a0f9a4fca3: Status 404 returned error can't find the container with id aad5999b2d7394cb6d472a5eee688eedf160804ca8692bd5025e41a0f9a4fca3
Nov 03 12:35:02.475621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.475589    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh" event={"ID":"282ce652-1641-4658-a189-eeaff2904167","Type":"ContainerStarted","Data":"1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba"}
Nov 03 12:35:02.475806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.475648    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh" event={"ID":"282ce652-1641-4658-a189-eeaff2904167","Type":"ContainerStarted","Data":"aad5999b2d7394cb6d472a5eee688eedf160804ca8692bd5025e41a0f9a4fca3"}
Nov 03 12:35:02.475806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.475679    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:02.477369 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.477339    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp" event={"ID":"11b73485-b056-4b49-991f-9be46681cba8","Type":"ContainerStarted","Data":"324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b"}
Nov 03 12:35:02.477492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.477392    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:02.477492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.477416    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp" event={"ID":"11b73485-b056-4b49-991f-9be46681cba8","Type":"ContainerStarted","Data":"016676204fd84b7a0a92d01c43db21c0e00f5710d15d2f296225b4ab95e21e54"}
Nov 03 12:35:02.581758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.581694    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh" podStartSLOduration=1.581679836 podStartE2EDuration="1.581679836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:35:02.579114171 +0000 UTC m=+2651.591580834" watchObservedRunningTime="2026-11-03 12:35:02.581679836 +0000 UTC m=+2651.594146498"
Nov 03 12:35:02.687901 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:02.687853    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp" podStartSLOduration=1.687836135 podStartE2EDuration="1.687836135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:35:02.682551545 +0000 UTC m=+2651.695018209" watchObservedRunningTime="2026-11-03 12:35:02.687836135 +0000 UTC m=+2651.700302797"
Nov 03 12:35:09.564064 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:09.564031    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:35:13.487355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:13.487318    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:35:13.487833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:13.487393    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:35:23.926607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:23.926575    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"]
Nov 03 12:35:23.927112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:23.926801    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy" containerID="cri-o://880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f" gracePeriod=30
Nov 03 12:35:25.330899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:25.330866    4909 patch_prober.go:29] interesting pod/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:35:25.331393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:25.330925    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:35:29.207178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.207153    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:35:29.305288 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305259    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-podinfo\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.305441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305306    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-envoy\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.305441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305349    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-data\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.305529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305451    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-token\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-token\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.305573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305544    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-socket\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.305614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305580    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b95018a-6308-4bb1-b42d-3ac6f472a936-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b95018a-6308-4bb1-b42d-3ac6f472a936-istiod-ca-cert\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.305677 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305661    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-credential-socket\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.306088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305759    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-data" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:29.306088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305784    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-kube-api-access-2tnsl\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-kube-api-access-2tnsl\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.306088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305841    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-certs\") pod \"9b95018a-6308-4bb1-b42d-3ac6f472a936\" (UID: \"9b95018a-6308-4bb1-b42d-3ac6f472a936\") "
Nov 03 12:35:29.306088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305896    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-socket" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:29.306088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.305982    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-credential-socket" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:29.306088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.306057    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b95018a-6308-4bb1-b42d-3ac6f472a936-istiod-ca-cert" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:29.306494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.306178    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-certs" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:29.306494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.306365    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.306494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.306387    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b95018a-6308-4bb1-b42d-3ac6f472a936-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.306494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.306403    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.306494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.306421    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.306494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.306435    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.307558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.307534    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-envoy" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:29.307633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.307583    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-podinfo" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:29.307822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.307804    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-token" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:29.308011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.307998    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-kube-api-access-2tnsl" pod "9b95018a-6308-4bb1-b42d-3ac6f472a936" (UID: "9b95018a-6308-4bb1-b42d-3ac6f472a936"). InnerVolumeSpecName "kube-api-access-2tnsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:29.435723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.428460    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.435723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.428501    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.435723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.428516    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.435723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.428533    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tnsl\" (UniqueName: \"kubernetes.io/projected/9b95018a-6308-4bb1-b42d-3ac6f472a936-kube-api-access-2tnsl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:29.661918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.661834    4909 generic.go:356] "Generic (PLEG): container finished" podID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerID="880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f" exitCode=0
Nov 03 12:35:29.661918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.661888    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"
Nov 03 12:35:29.662149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.661916    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" event={"ID":"9b95018a-6308-4bb1-b42d-3ac6f472a936","Type":"ContainerDied","Data":"880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f"}
Nov 03 12:35:29.662149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.661972    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4" event={"ID":"9b95018a-6308-4bb1-b42d-3ac6f472a936","Type":"ContainerDied","Data":"9d459acb39c38a4735b9e097146636bc2b6b95413818f16100c789083385a1e5"}
Nov 03 12:35:29.662149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.661994    4909 scope.go:122] "RemoveContainer" containerID="880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f"
Nov 03 12:35:29.673226 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.673205    4909 scope.go:122] "RemoveContainer" containerID="b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651"
Nov 03 12:35:29.688416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.688389    4909 scope.go:122] "RemoveContainer" containerID="880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f"
Nov 03 12:35:29.688712 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:29.688679    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f\": container with ID starting with 880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f not found: ID does not exist" containerID="880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f"
Nov 03 12:35:29.688811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.688738    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f"} err="failed to get container status \"880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f\": rpc error: code = NotFound desc = could not find container \"880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f\": container with ID starting with 880aea6c9ff388d61f0712625eaef3f4d91f42d92e6392a90f9aac5369ac7e7f not found: ID does not exist"
Nov 03 12:35:29.688811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.688761    4909 scope.go:122] "RemoveContainer" containerID="b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651"
Nov 03 12:35:29.689053 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:29.689028    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651\": container with ID starting with b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651 not found: ID does not exist" containerID="b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651"
Nov 03 12:35:29.689145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.689056    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651"} err="failed to get container status \"b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651\": rpc error: code = NotFound desc = could not find container \"b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651\": container with ID starting with b2c2107af325f84dbff42a545e6ebb4ce4741bff2463e00744f2072a59fea651 not found: ID does not exist"
Nov 03 12:35:29.704389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.704366    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:35:29.815192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.814912    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"]
Nov 03 12:35:29.818108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:29.818088    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb4a0f-openshift-default-5d4c99d84f-jd5v4"]
Nov 03 12:35:31.125711 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:31.125678    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" path="/var/lib/kubelet/pods/9b95018a-6308-4bb1-b42d-3ac6f472a936/volumes"
Nov 03 12:35:36.383643 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:36.383599    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"]
Nov 03 12:35:36.483242 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:36.483207    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"]
Nov 03 12:35:38.886261 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:38.886225    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"]
Nov 03 12:35:38.886739 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:38.886432    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w" podUID="3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" containerName="maas-api" containerID="cri-o://e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4" gracePeriod=30
Nov 03 12:35:39.207531 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.207505    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:35:39.400238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.400195    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-kube-api-access-5tw62\" (UniqueName: \"kubernetes.io/projected/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-kube-api-access-5tw62\") pod \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") "
Nov 03 12:35:39.400418 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.400283    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-openshift-service-ca\") pod \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") "
Nov 03 12:35:39.400418 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.400336    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls\") pod \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") "
Nov 03 12:35:39.400418 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.400399    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-metrics-certs\") pod \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\" (UID: \"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c\") "
Nov 03 12:35:39.400686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.400660    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-openshift-service-ca" pod "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" (UID: "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:39.402517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.402496    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-metrics-certs" pod "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" (UID: "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:35:39.402596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.402547    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls" pod "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" (UID: "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:35:39.402596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.402583    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-kube-api-access-5tw62" pod "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" (UID: "3d41305d-7ab4-4011-ada5-72bdd4cb9d2c"). InnerVolumeSpecName "kube-api-access-5tw62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:39.501919 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.501886    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tw62\" (UniqueName: \"kubernetes.io/projected/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-kube-api-access-5tw62\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:39.501919 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.501915    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:39.501919 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.501926    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:39.502244 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.501934    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:39.565127 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:39.565091    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:35:39.728664 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.728628    4909 generic.go:356] "Generic (PLEG): container finished" podID="3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" containerID="e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4" exitCode=0
Nov 03 12:35:39.728881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.728669    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w" event={"ID":"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c","Type":"ContainerDied","Data":"e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4"}
Nov 03 12:35:39.728881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.728684    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"
Nov 03 12:35:39.728881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.728798    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w" event={"ID":"3d41305d-7ab4-4011-ada5-72bdd4cb9d2c","Type":"ContainerDied","Data":"db830a220127d841e66683f351aa8e3dbd90759518fc5deb2da0dd285e6ababa"}
Nov 03 12:35:39.728881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.728832    4909 scope.go:122] "RemoveContainer" containerID="e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4"
Nov 03 12:35:39.739630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.739613    4909 scope.go:122] "RemoveContainer" containerID="e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4"
Nov 03 12:35:39.739898 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:39.739880    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4\": container with ID starting with e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4 not found: ID does not exist" containerID="e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4"
Nov 03 12:35:39.739954 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.739905    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4"} err="failed to get container status \"e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4\": rpc error: code = NotFound desc = could not find container \"e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4\": container with ID starting with e00f24bcbbc4da166570a73a6bf3bf77e12934f0206c988ea66518b73634e9e4 not found: ID does not exist"
Nov 03 12:35:39.907719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.907577    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"]
Nov 03 12:35:39.925258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:39.925221    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbca30-5b58c7d4c6-6qg9w"]
Nov 03 12:35:40.429229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429198    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"]
Nov 03 12:35:40.429831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429816    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy"
Nov 03 12:35:40.429887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429831    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy"
Nov 03 12:35:40.429887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429851    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" containerName="maas-api"
Nov 03 12:35:40.429887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429857    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" containerName="maas-api"
Nov 03 12:35:40.429887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429878    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy"
Nov 03 12:35:40.429887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429883    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy"
Nov 03 12:35:40.430043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429972    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy"
Nov 03 12:35:40.430043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429987    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b95018a-6308-4bb1-b42d-3ac6f472a936" containerName="istio-proxy"
Nov 03 12:35:40.430043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.429996    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" containerName="maas-api"
Nov 03 12:35:40.435376 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.435359    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.483792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.483754    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e0615c-serving-cert\""
Nov 03 12:35:40.487304 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.487284    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"]
Nov 03 12:35:40.514978 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.514954    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.515113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.515094    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.515214 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.515197    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xlj\" (UniqueName: \"kubernetes.io/projected/53e5714c-34d7-4250-8aad-724b2915baf7-kube-api-access-z9xlj\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.515292 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.515229    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e5714c-34d7-4250-8aad-724b2915baf7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.616154 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.616117    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.616543 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.616516    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.616666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.616624    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xlj\" (UniqueName: \"kubernetes.io/projected/53e5714c-34d7-4250-8aad-724b2915baf7-kube-api-access-z9xlj\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.616666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.616645    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e5714c-34d7-4250-8aad-724b2915baf7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.616666 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:40.616649    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-serving-cert: secret "maas-api-e2e-shared-b-w2-e0615c-serving-cert" not found
Nov 03 12:35:40.616856 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:40.616735    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls podName:53e5714c-34d7-4250-8aad-724b2915baf7 nodeName:}" failed. No retries permitted until 2026-11-03 12:35:41.116714139 +0000 UTC m=+2690.129180823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls") pod "maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" (UID: "53e5714c-34d7-4250-8aad-724b2915baf7") : secret "maas-api-e2e-shared-b-w2-e0615c-serving-cert" not found
Nov 03 12:35:40.617458 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.617431    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e5714c-34d7-4250-8aad-724b2915baf7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.618729 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.618708    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:40.723718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:40.709591    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xlj\" (UniqueName: \"kubernetes.io/projected/53e5714c-34d7-4250-8aad-724b2915baf7-kube-api-access-z9xlj\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:41.125332 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.125243    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d41305d-7ab4-4011-ada5-72bdd4cb9d2c" path="/var/lib/kubelet/pods/3d41305d-7ab4-4011-ada5-72bdd4cb9d2c/volumes"
Nov 03 12:35:41.131686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.131659    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:41.134155 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.134134    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:41.349473 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.349444    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:41.384585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.384505    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp" podUID="24244784-15a7-442c-a408-b132052c07df" containerName="payload-processing" containerID="cri-o://c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375" gracePeriod=25
Nov 03 12:35:41.484583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.484367    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss" podUID="844132c2-044e-402d-9e2a-92c829b6f0bd" containerName="payload-pre-processing" containerID="cri-o://93a10ccbbc25416df531aeccae7d221f6c196c887059aa148d3ac94018f9e4d7" gracePeriod=25
Nov 03 12:35:41.688940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.688915    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:35:41.742519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.742493    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"]
Nov 03 12:35:41.832737 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:35:41.796307    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e5714c_34d7_4250_8aad_724b2915baf7.slice/crio-30b94cde149d490707209a3b81064cfc7c1f8a027c7096d2bf1060de1aef3efe WatchSource:0}: Error finding container 30b94cde149d490707209a3b81064cfc7c1f8a027c7096d2bf1060de1aef3efe: Status 404 returned error can't find the container with id 30b94cde149d490707209a3b81064cfc7c1f8a027c7096d2bf1060de1aef3efe
Nov 03 12:35:41.911465 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.911385    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24244784-15a7-442c-a408-b132052c07df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24244784-15a7-442c-a408-b132052c07df-plugins-config-volume\") pod \"24244784-15a7-442c-a408-b132052c07df\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") "
Nov 03 12:35:41.915070 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.915045    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24244784-15a7-442c-a408-b132052c07df-kube-api-access-ph7cv\" (UniqueName: \"kubernetes.io/projected/24244784-15a7-442c-a408-b132052c07df-kube-api-access-ph7cv\") pod \"24244784-15a7-442c-a408-b132052c07df\" (UID: \"24244784-15a7-442c-a408-b132052c07df\") "
Nov 03 12:35:41.916128 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.914876    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24244784-15a7-442c-a408-b132052c07df-plugins-config-volume" pod "24244784-15a7-442c-a408-b132052c07df" (UID: "24244784-15a7-442c-a408-b132052c07df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:41.934007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.933906    4909 generic.go:356] "Generic (PLEG): container finished" podID="24244784-15a7-442c-a408-b132052c07df" containerID="c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375" exitCode=0
Nov 03 12:35:41.934007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.933966    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"
Nov 03 12:35:41.934007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.933991    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp" event={"ID":"24244784-15a7-442c-a408-b132052c07df","Type":"ContainerDied","Data":"c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375"}
Nov 03 12:35:41.934248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.934059    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp" event={"ID":"24244784-15a7-442c-a408-b132052c07df","Type":"ContainerDied","Data":"5f1df2703cb790a0fa02fc1a7f20ddf1a46fb63a6974aa6806859ee79504e33d"}
Nov 03 12:35:41.934248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.934091    4909 scope.go:122] "RemoveContainer" containerID="c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375"
Nov 03 12:35:41.938961 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.938931    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24244784-15a7-442c-a408-b132052c07df-kube-api-access-ph7cv" pod "24244784-15a7-442c-a408-b132052c07df" (UID: "24244784-15a7-442c-a408-b132052c07df"). InnerVolumeSpecName "kube-api-access-ph7cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:41.939484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.939465    4909 generic.go:356] "Generic (PLEG): container finished" podID="844132c2-044e-402d-9e2a-92c829b6f0bd" containerID="93a10ccbbc25416df531aeccae7d221f6c196c887059aa148d3ac94018f9e4d7" exitCode=0
Nov 03 12:35:41.939599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.939580    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss" event={"ID":"844132c2-044e-402d-9e2a-92c829b6f0bd","Type":"ContainerDied","Data":"93a10ccbbc25416df531aeccae7d221f6c196c887059aa148d3ac94018f9e4d7"}
Nov 03 12:35:41.952904 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.952865    4909 scope.go:122] "RemoveContainer" containerID="c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375"
Nov 03 12:35:41.953184 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:41.953164    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375\": container with ID starting with c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375 not found: ID does not exist" containerID="c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375"
Nov 03 12:35:41.953327 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.953196    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375"} err="failed to get container status \"c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375\": rpc error: code = NotFound desc = could not find container \"c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375\": container with ID starting with c672ddfde0429fedd8f92bf1ad47eaeb91b15cf93494aa055a40909799e35375 not found: ID does not exist"
Nov 03 12:35:41.954906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:41.954858    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:35:42.017106 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.017061    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844132c2-044e-402d-9e2a-92c829b6f0bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844132c2-044e-402d-9e2a-92c829b6f0bd-plugins-config-volume\") pod \"844132c2-044e-402d-9e2a-92c829b6f0bd\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") "
Nov 03 12:35:42.017353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.017319    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844132c2-044e-402d-9e2a-92c829b6f0bd-kube-api-access-bwk5h\" (UniqueName: \"kubernetes.io/projected/844132c2-044e-402d-9e2a-92c829b6f0bd-kube-api-access-bwk5h\") pod \"844132c2-044e-402d-9e2a-92c829b6f0bd\" (UID: \"844132c2-044e-402d-9e2a-92c829b6f0bd\") "
Nov 03 12:35:42.017485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.017459    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844132c2-044e-402d-9e2a-92c829b6f0bd-plugins-config-volume" pod "844132c2-044e-402d-9e2a-92c829b6f0bd" (UID: "844132c2-044e-402d-9e2a-92c829b6f0bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:42.017892 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.017872    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844132c2-044e-402d-9e2a-92c829b6f0bd-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:42.017989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.017902    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24244784-15a7-442c-a408-b132052c07df-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:42.017989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.017917    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7cv\" (UniqueName: \"kubernetes.io/projected/24244784-15a7-442c-a408-b132052c07df-kube-api-access-ph7cv\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:42.019734 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.019688    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844132c2-044e-402d-9e2a-92c829b6f0bd-kube-api-access-bwk5h" pod "844132c2-044e-402d-9e2a-92c829b6f0bd" (UID: "844132c2-044e-402d-9e2a-92c829b6f0bd"). InnerVolumeSpecName "kube-api-access-bwk5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:42.119803 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.119771    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwk5h\" (UniqueName: \"kubernetes.io/projected/844132c2-044e-402d-9e2a-92c829b6f0bd-kube-api-access-bwk5h\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:42.347068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.347036    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"]
Nov 03 12:35:42.350608 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.350584    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbca30-695b6cd48c-f2rnp"]
Nov 03 12:35:42.948280 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.948249    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"
Nov 03 12:35:42.948429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.948281    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss" event={"ID":"844132c2-044e-402d-9e2a-92c829b6f0bd","Type":"ContainerDied","Data":"2a084cfed5dfcc40de2b28234f18f0f619bc257bfb42507b365721a8d075313c"}
Nov 03 12:35:42.948429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.948342    4909 scope.go:122] "RemoveContainer" containerID="93a10ccbbc25416df531aeccae7d221f6c196c887059aa148d3ac94018f9e4d7"
Nov 03 12:35:42.950378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.950351    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" event={"ID":"53e5714c-34d7-4250-8aad-724b2915baf7","Type":"ContainerStarted","Data":"6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412"}
Nov 03 12:35:42.950528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.950402    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" event={"ID":"53e5714c-34d7-4250-8aad-724b2915baf7","Type":"ContainerStarted","Data":"30b94cde149d490707209a3b81064cfc7c1f8a027c7096d2bf1060de1aef3efe"}
Nov 03 12:35:42.950623 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:42.950604    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:43.104010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.103963    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"]
Nov 03 12:35:43.123657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.123626    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24244784-15a7-442c-a408-b132052c07df" path="/var/lib/kubelet/pods/24244784-15a7-442c-a408-b132052c07df/volumes"
Nov 03 12:35:43.143550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.143510    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbca30-5ddb6fd7c9-nnsss"]
Nov 03 12:35:43.207428 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.207351    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"]
Nov 03 12:35:43.207604 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.207555    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy" containerID="cri-o://ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03" gracePeriod=30
Nov 03 12:35:43.217131 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.217092    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"]
Nov 03 12:35:43.217667 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.217335    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy" containerID="cri-o://765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747" gracePeriod=30
Nov 03 12:35:43.254633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:43.254589    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" podStartSLOduration=2.399608595 podStartE2EDuration="3.254576101s" totalImagesPullingTime="854.967506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:35:43.252324815 +0000 UTC m=+2692.264791481" watchObservedRunningTime="2026-11-03 12:35:43.254576101 +0000 UTC m=+2692.267042764"
Nov 03 12:35:45.124816 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:45.124774    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844132c2-044e-402d-9e2a-92c829b6f0bd" path="/var/lib/kubelet/pods/844132c2-044e-402d-9e2a-92c829b6f0bd/volumes"
Nov 03 12:35:48.499672 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.499643    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:35:48.524417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.524397    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:35:48.606251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606220    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-token\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-token\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.606422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606260    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-data\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.606422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606308    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-socket\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.606422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606341    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-podinfo\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.606422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606396    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36527505-c91a-4919-af7c-24a516e2fc81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36527505-c91a-4919-af7c-24a516e2fc81-istio-podinfo\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.606650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606443    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-data\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.606650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606481    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36527505-c91a-4919-af7c-24a516e2fc81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36527505-c91a-4919-af7c-24a516e2fc81-istiod-ca-cert\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.606650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606542    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-kube-api-access-895wc\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-kube-api-access-895wc\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.606650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606573    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-certs\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.606650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606630    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-socket" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.606650 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606647    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-data" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.607016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606734    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-credential-socket\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.607016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606778    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-envoy\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.607016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606780    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-data" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.607016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606819    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-certs\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.607016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606871    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istiod-ca-cert\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.607016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.606900    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-certs" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.607016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607007    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-credential-socket" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.607484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607062    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36527505-c91a-4919-af7c-24a516e2fc81-istiod-ca-cert" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:48.607484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607272    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istiod-ca-cert" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:48.607639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607604    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-istio-token\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-istio-token\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.607940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607648    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-socket\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.607940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607766    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-kube-api-access-bxxfs\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-kube-api-access-bxxfs\") pod \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\" (UID: \"5c6361dd-8e6f-4669-acb7-e586f2dedde4\") "
Nov 03 12:35:48.607940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607823    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-envoy\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.607940 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607866    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-credential-socket\") pod \"36527505-c91a-4919-af7c-24a516e2fc81\" (UID: \"36527505-c91a-4919-af7c-24a516e2fc81\") "
Nov 03 12:35:48.608235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.607951    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-certs" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.608290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608238    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-socket" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608533    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608562    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608577    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608591    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36527505-c91a-4919-af7c-24a516e2fc81-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608606    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608621    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608636    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608650    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.609011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.608664    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.610603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.610250    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-kube-api-access-895wc" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "kube-api-access-895wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.610603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.610533    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-credential-socket" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.611117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.611079    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-envoy" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.611378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.611302    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-podinfo" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:48.611378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.611307    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-istio-token" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.611378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.611361    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-token" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.611564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.611531    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36527505-c91a-4919-af7c-24a516e2fc81-istio-podinfo" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:48.612532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.612506    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-envoy" pod "36527505-c91a-4919-af7c-24a516e2fc81" (UID: "36527505-c91a-4919-af7c-24a516e2fc81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.612965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.612944    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-kube-api-access-bxxfs" pod "5c6361dd-8e6f-4669-acb7-e586f2dedde4" (UID: "5c6361dd-8e6f-4669-acb7-e586f2dedde4"). InnerVolumeSpecName "kube-api-access-bxxfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.710437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710399    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710425    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36527505-c91a-4919-af7c-24a516e2fc81-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710437 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710436    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895wc\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-kube-api-access-895wc\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710447    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710456    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36527505-c91a-4919-af7c-24a516e2fc81-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710465    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxxfs\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-kube-api-access-bxxfs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710477    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710488    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36527505-c91a-4919-af7c-24a516e2fc81-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.710737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.710498    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6361dd-8e6f-4669-acb7-e586f2dedde4-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.963688 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.963658    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 12:35:48.995125 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.995097    4909 generic.go:356] "Generic (PLEG): container finished" podID="36527505-c91a-4919-af7c-24a516e2fc81" containerID="ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03" exitCode=0
Nov 03 12:35:48.995270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.995130    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" event={"ID":"36527505-c91a-4919-af7c-24a516e2fc81","Type":"ContainerDied","Data":"ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03"}
Nov 03 12:35:48.995270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.995181    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7" event={"ID":"36527505-c91a-4919-af7c-24a516e2fc81","Type":"ContainerDied","Data":"42692fb225182aed288d7c350ed5e0b20f4f431c81e9998782f8712d73db0a3f"}
Nov 03 12:35:48.995270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.995210    4909 scope.go:122] "RemoveContainer" containerID="ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03"
Nov 03 12:35:48.995270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.995137    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"
Nov 03 12:35:48.996620 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.996603    4909 generic.go:356] "Generic (PLEG): container finished" podID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerID="765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747" exitCode=0
Nov 03 12:35:48.996732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.996660    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"
Nov 03 12:35:48.996732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.996681    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" event={"ID":"5c6361dd-8e6f-4669-acb7-e586f2dedde4","Type":"ContainerDied","Data":"765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747"}
Nov 03 12:35:48.996842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:48.996749    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn" event={"ID":"5c6361dd-8e6f-4669-acb7-e586f2dedde4","Type":"ContainerDied","Data":"7c0d62fe85c9166af58fe5fc762088492882fa47dbc47d99caa40efaebb60c9b"}
Nov 03 12:35:49.008353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.008336    4909 scope.go:122] "RemoveContainer" containerID="ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03"
Nov 03 12:35:49.008596 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:49.008579    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03\": container with ID starting with ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03 not found: ID does not exist" containerID="ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03"
Nov 03 12:35:49.008672 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.008606    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03"} err="failed to get container status \"ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03\": rpc error: code = NotFound desc = could not find container \"ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03\": container with ID starting with ea496d0161528c60be853c9dfddafdf4e163216da12b15ebc8d9be6ef7101a03 not found: ID does not exist"
Nov 03 12:35:49.008672 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.008629    4909 scope.go:122] "RemoveContainer" containerID="765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747"
Nov 03 12:35:49.018248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.018232    4909 scope.go:122] "RemoveContainer" containerID="765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747"
Nov 03 12:35:49.018507 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:35:49.018486    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747\": container with ID starting with 765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747 not found: ID does not exist" containerID="765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747"
Nov 03 12:35:49.018573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.018513    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747"} err="failed to get container status \"765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747\": rpc error: code = NotFound desc = could not find container \"765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747\": container with ID starting with 765c9fcfcb047039c95403f926a340092cdd5334c692e1bded06470382c55747 not found: ID does not exist"
Nov 03 12:35:49.175097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.175055    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"]
Nov 03 12:35:49.179027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.179003    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tnjz7"]
Nov 03 12:35:49.279546 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.279506    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"]
Nov 03 12:35:49.285296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:49.285269    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-rmngn"]
Nov 03 12:35:51.124237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:51.124208    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36527505-c91a-4919-af7c-24a516e2fc81" path="/var/lib/kubelet/pods/36527505-c91a-4919-af7c-24a516e2fc81/volumes"
Nov 03 12:35:51.124687 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:51.124673    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" path="/var/lib/kubelet/pods/5c6361dd-8e6f-4669-acb7-e586f2dedde4/volumes"
Nov 03 12:35:51.465752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:51.465722    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:35:51.468929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:51.468907    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:35:52.925602 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.925570    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"]
Nov 03 12:35:52.926250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926233    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy"
Nov 03 12:35:52.926312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926251    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy"
Nov 03 12:35:52.926312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926271    4909 cpu_manager.go:414] "containerMap: removing container" podUID="844132c2-044e-402d-9e2a-92c829b6f0bd" containerName="payload-pre-processing"
Nov 03 12:35:52.926312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926277    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844132c2-044e-402d-9e2a-92c829b6f0bd" containerName="payload-pre-processing"
Nov 03 12:35:52.926312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926286    4909 cpu_manager.go:414] "containerMap: removing container" podUID="24244784-15a7-442c-a408-b132052c07df" containerName="payload-processing"
Nov 03 12:35:52.926312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926291    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24244784-15a7-442c-a408-b132052c07df" containerName="payload-processing"
Nov 03 12:35:52.926312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926310    4909 cpu_manager.go:414] "containerMap: removing container" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy"
Nov 03 12:35:52.926495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926315    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy"
Nov 03 12:35:52.926495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926394    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="844132c2-044e-402d-9e2a-92c829b6f0bd" containerName="payload-pre-processing"
Nov 03 12:35:52.926495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926403    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="24244784-15a7-442c-a408-b132052c07df" containerName="payload-processing"
Nov 03 12:35:52.926495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926414    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="36527505-c91a-4919-af7c-24a516e2fc81" containerName="istio-proxy"
Nov 03 12:35:52.926495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.926424    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6361dd-8e6f-4669-acb7-e586f2dedde4" containerName="istio-proxy"
Nov 03 12:35:52.931736 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.931692    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:52.985890 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:52.985865    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-561bd4-openshift-default-dockercfg-h9hnf\""
Nov 03 12:35:53.032951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.032915    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"]
Nov 03 12:35:53.063116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063074    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-socket\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063124    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-credential-socket\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063153    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6f6\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-kube-api-access-mj6f6\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063204    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istiod-ca-cert\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063264    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-podinfo\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063292    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-data\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063331    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-token\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063439    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-certs\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.063840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.063537    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-envoy\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.172863    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-socket\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.172928    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-credential-socket\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.172958    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6f6\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-kube-api-access-mj6f6\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.173018    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istiod-ca-cert\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.173090    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-podinfo\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.173124    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-data\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.173171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-token\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.173298    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-certs\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.178769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.173421    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-envoy\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.189720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.180114    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istiod-ca-cert\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.189720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.180387    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-data\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.189720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.180404    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-socket\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.189720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.180639    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-credential-socket\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.189720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.181110    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-certs\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.189720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.183761    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-envoy\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.210038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.191016    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-podinfo\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.235564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.235531    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6f6\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-kube-api-access-mj6f6\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.235754 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.235728    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-token\") pod \"e2e-autores-561bd4-openshift-default-747678d87f-nn4lx\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") " pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.244748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.244725    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:53.695034 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.695005    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"]
Nov 03 12:35:53.696128 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:35:53.696101    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4e83f6_33a6_411e_87e9_a31bb4ab0b6d.slice/crio-29fac1dcead3ebd58b49a5ee26b086d0c5c4405e626f9f42251c89cc0ef9a7a2 WatchSource:0}: Error finding container 29fac1dcead3ebd58b49a5ee26b086d0c5c4405e626f9f42251c89cc0ef9a7a2: Status 404 returned error can't find the container with id 29fac1dcead3ebd58b49a5ee26b086d0c5c4405e626f9f42251c89cc0ef9a7a2
Nov 03 12:35:53.698771 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.698741    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:35:53.698866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.698798    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:35:53.698866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:53.698827    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:35:54.050044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:54.049944    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" event={"ID":"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d","Type":"ContainerStarted","Data":"643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21"}
Nov 03 12:35:54.050044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:54.050023    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" event={"ID":"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d","Type":"ContainerStarted","Data":"29fac1dcead3ebd58b49a5ee26b086d0c5c4405e626f9f42251c89cc0ef9a7a2"}
Nov 03 12:35:54.160442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:54.160393    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podStartSLOduration=2.160379393 podStartE2EDuration="2.160379393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:35:54.157887432 +0000 UTC m=+2703.170354095" watchObservedRunningTime="2026-11-03 12:35:54.160379393 +0000 UTC m=+2703.172846056"
Nov 03 12:35:54.244837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:54.244805    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:54.250170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:54.250145    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:55.057555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:55.057526    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:55.058371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:55.058349    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:35:57.556380 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:35:57.556333    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:36:01.002012 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.001977    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"]
Nov 03 12:36:01.007522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.007501    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.030045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.030022    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.030154 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.030080    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl24s\" (UniqueName: \"kubernetes.io/projected/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-kube-api-access-bl24s\") pod \"payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.087589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.087550    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"]
Nov 03 12:36:01.209723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.195178    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-561bd4\""
Nov 03 12:36:01.209723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.195394    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-561bd4-dockercfg-bdzgb\""
Nov 03 12:36:01.249782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.226966    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.249782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.227096    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl24s\" (UniqueName: \"kubernetes.io/projected/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-kube-api-access-bl24s\") pod \"payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.249782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.229348    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.289794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.289717    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl24s\" (UniqueName: \"kubernetes.io/projected/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-kube-api-access-bl24s\") pod \"payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.290233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.290209    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"]
Nov 03 12:36:01.297038 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.297021    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"]
Nov 03 12:36:01.297238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.297200    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.322630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.322611    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:01.328883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.328860    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgt6\" (UniqueName: \"kubernetes.io/projected/184adf72-7527-4b4e-a053-b691555cff90-kube-api-access-9tgt6\") pod \"payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") " pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.329007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.328926    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184adf72-7527-4b4e-a053-b691555cff90-plugins-config-volume\") pod \"payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") " pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.476294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.474677    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgt6\" (UniqueName: \"kubernetes.io/projected/184adf72-7527-4b4e-a053-b691555cff90-kube-api-access-9tgt6\") pod \"payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") " pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.476294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.474771    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184adf72-7527-4b4e-a053-b691555cff90-plugins-config-volume\") pod \"payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") " pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.476294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.475863    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184adf72-7527-4b4e-a053-b691555cff90-plugins-config-volume\") pod \"payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") " pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.530754 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.529799    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgt6\" (UniqueName: \"kubernetes.io/projected/184adf72-7527-4b4e-a053-b691555cff90-kube-api-access-9tgt6\") pod \"payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") " pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.619317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.619236    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:01.733922 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:01.729068    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"]
Nov 03 12:36:02.022295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.022263    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"]
Nov 03 12:36:02.023349 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:36:02.023317    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184adf72_7527_4b4e_a053_b691555cff90.slice/crio-f6907c7f936625b7aa70b2070a0e7718cabbefdbbb8a35eafc848c9812e028e8 WatchSource:0}: Error finding container f6907c7f936625b7aa70b2070a0e7718cabbefdbbb8a35eafc848c9812e028e8: Status 404 returned error can't find the container with id f6907c7f936625b7aa70b2070a0e7718cabbefdbbb8a35eafc848c9812e028e8
Nov 03 12:36:02.287385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.287291    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng" event={"ID":"184adf72-7527-4b4e-a053-b691555cff90","Type":"ContainerStarted","Data":"0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702"}
Nov 03 12:36:02.287385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.287365    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng" event={"ID":"184adf72-7527-4b4e-a053-b691555cff90","Type":"ContainerStarted","Data":"f6907c7f936625b7aa70b2070a0e7718cabbefdbbb8a35eafc848c9812e028e8"}
Nov 03 12:36:02.287617 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.287509    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:02.289116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.289092    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq" event={"ID":"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2","Type":"ContainerStarted","Data":"d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7"}
Nov 03 12:36:02.289253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.289129    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq" event={"ID":"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2","Type":"ContainerStarted","Data":"c340dad41c85e91dd1e577e5f1bc1eb1aa3e529a2356a47d1f9ed4a90744398f"}
Nov 03 12:36:02.289253 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.289199    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:02.394219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.394168    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng" podStartSLOduration=1.394154365 podStartE2EDuration="1.394154365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:36:02.391075321 +0000 UTC m=+2711.403541983" watchObservedRunningTime="2026-11-03 12:36:02.394154365 +0000 UTC m=+2711.406621027"
Nov 03 12:36:02.502689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:02.502626    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq" podStartSLOduration=2.502609121 podStartE2EDuration="2.502609121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:36:02.501983068 +0000 UTC m=+2711.514449729" watchObservedRunningTime="2026-11-03 12:36:02.502609121 +0000 UTC m=+2711.515075785"
Nov 03 12:36:08.331075 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:08.331043    4909 generic.go:356] "Generic (PLEG): container finished" podID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerID="b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd" exitCode=0
Nov 03 12:36:08.331595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:08.331116    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" event={"ID":"f8698500-d104-43a2-bb62-aae06cc3a4df","Type":"ContainerDied","Data":"b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd"}
Nov 03 12:36:08.331595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:08.331538    4909 scope.go:122] "RemoveContainer" containerID="b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd"
Nov 03 12:36:08.333201 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:08.333167    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:36:08.333291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:08.333258    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:36:08.333333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:08.333307    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:36:09.344722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:09.339955    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" event={"ID":"f8698500-d104-43a2-bb62-aae06cc3a4df","Type":"ContainerStarted","Data":"b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6"}
Nov 03 12:36:09.587623 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:36:09.587586    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:36:09.961842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:09.961806    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:36:10.019062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:10.019027    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:36:10.020405 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:10.020381    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 12:36:10.020491 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:10.020432    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 12:36:11.019043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:11.019005    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 12:36:11.019538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:11.019063    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 12:36:12.018589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:12.018553    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 12:36:12.018882 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:12.018606    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 12:36:13.018566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:13.018519    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 12:36:13.018994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:13.018582    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 12:36:13.304215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:13.304134    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:36:13.304500 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:13.304482    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:36:14.019062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:14.019025    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 12:36:14.019492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:14.019089    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 12:36:15.023192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:15.023157    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:36:15.024295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:15.024277    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 12:36:24.366328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.366292    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:36:24.445907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.445875    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"]
Nov 03 12:36:24.450300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.450284    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.500453 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.500427    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"openshift-service-ca.crt\""
Nov 03 12:36:24.500998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.500662    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"kube-root-ca.crt\""
Nov 03 12:36:24.501837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.501818    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"default-dockercfg-rp9tp\""
Nov 03 12:36:24.522653 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.522625    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-tmp-dir\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.522795 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.522660    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-model-cache\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.522871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.522795    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-home\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.522871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.522830    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-dshm\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.522991 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.522904    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stx9b\" (UniqueName: \"kubernetes.io/projected/a2d08d98-334b-4bc2-bb65-f4e566bada67-kube-api-access-stx9b\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.522991 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.522931    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d08d98-334b-4bc2-bb65-f4e566bada67-tls-certs\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.545920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.545896    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"auth-test-model-1250b3-kserve-self-signed-certs\""
Nov 03 12:36:24.549235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.549216    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"]
Nov 03 12:36:24.624355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624280    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stx9b\" (UniqueName: \"kubernetes.io/projected/a2d08d98-334b-4bc2-bb65-f4e566bada67-kube-api-access-stx9b\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.624355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624320    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d08d98-334b-4bc2-bb65-f4e566bada67-tls-certs\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.624532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624440    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-tmp-dir\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.624532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624466    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-model-cache\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.624612 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624565    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-home\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.624612 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624595    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-dshm\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.624956 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624927    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-tmp-dir\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.625060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624938    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-model-cache\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.625060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.624977    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-home\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.626908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.626885    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-dshm\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.627052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.627036    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d08d98-334b-4bc2-bb65-f4e566bada67-tls-certs\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.676692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.676663    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stx9b\" (UniqueName: \"kubernetes.io/projected/a2d08d98-334b-4bc2-bb65-f4e566bada67-kube-api-access-stx9b\") pod \"auth-test-model-1250b3-kserve-7866ccc67c-5npxj\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:24.763404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:24.763372    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:25.134094 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:25.134062    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"]
Nov 03 12:36:25.134741 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:36:25.134598    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d08d98_334b_4bc2_bb65_f4e566bada67.slice/crio-5dfd490b45ebc7d3882a3e069cee994b909ba03e417d05c332a1e45376733e0e WatchSource:0}: Error finding container 5dfd490b45ebc7d3882a3e069cee994b909ba03e417d05c332a1e45376733e0e: Status 404 returned error can't find the container with id 5dfd490b45ebc7d3882a3e069cee994b909ba03e417d05c332a1e45376733e0e
Nov 03 12:36:25.464888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:25.464856    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj" event={"ID":"a2d08d98-334b-4bc2-bb65-f4e566bada67","Type":"ContainerStarted","Data":"5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2"}
Nov 03 12:36:25.465357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:25.464899    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj" event={"ID":"a2d08d98-334b-4bc2-bb65-f4e566bada67","Type":"ContainerStarted","Data":"5dfd490b45ebc7d3882a3e069cee994b909ba03e417d05c332a1e45376733e0e"}
Nov 03 12:36:25.571748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:25.571601    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj" podStartSLOduration=1.571586157 podStartE2EDuration="1.571586157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:36:25.569839874 +0000 UTC m=+2734.582306536" watchObservedRunningTime="2026-11-03 12:36:25.571586157 +0000 UTC m=+2734.584052819"
Nov 03 12:36:25.763879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:25.763799    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:34.764384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:34.764302    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:34.776402 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:34.776381    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:35.541559 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:35.541513    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:36:39.565479 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:36:39.565443    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:36:57.635037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:36:57.635005    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:37:09.564864 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:37:09.564828    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:37:26.101832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:26.101795    4909 patch_prober.go:29] interesting pod/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:37:26.102264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:26.101856    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:37:27.661645 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:37:27.661613    4909 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4e83f6_33a6_411e_87e9_a31bb4ab0b6d.slice/crio-643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 12:37:27.902724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:27.902616    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerID="643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21" exitCode=0
Nov 03 12:37:27.902724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:27.902679    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" event={"ID":"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d","Type":"ContainerDied","Data":"643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21"}
Nov 03 12:37:27.903023 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:27.903008    4909 scope.go:122] "RemoveContainer" containerID="643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21"
Nov 03 12:37:27.904429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:27.904398    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:37:27.904510 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:27.904467    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:37:27.904510 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:27.904498    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:37:28.911759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:28.911718    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" event={"ID":"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d","Type":"ContainerStarted","Data":"a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d"}
Nov 03 12:37:29.259721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:29.247089    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:37:29.259721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:29.247243    4909 patch_prober.go:29] interesting pod/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:37:29.259721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:29.247289    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 12:37:30.249099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:30.248244    4909 patch_prober.go:29] interesting pod/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:37:30.249099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:30.248300    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 12:37:31.251338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:31.251287    4909 patch_prober.go:29] interesting pod/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:37:31.253033 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:31.253001    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 12:37:32.256732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:32.255198    4909 patch_prober.go:29] interesting pod/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:37:32.256732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:32.255269    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 12:37:33.255723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:33.248272    4909 patch_prober.go:29] interesting pod/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:37:33.255723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:33.248350    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 12:37:34.301978 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:34.301922    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:37:34.302637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:34.302613    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:37:34.303416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:34.303392    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:37:39.564850 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:37:39.564814    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:37:46.942113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:37:46.942076    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:38:05.048525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.048447    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"]
Nov 03 12:38:05.052973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.052954    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.107097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.107063    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"]
Nov 03 12:38:05.119788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.119735    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a4455f\"/\"openshift-service-ca.crt\""
Nov 03 12:38:05.123379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.123351    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-home\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.123521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.123402    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzg6v\" (UniqueName: \"kubernetes.io/projected/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-kube-api-access-fzg6v\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.123588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.123527    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tls-certs\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.123588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.123556    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-dshm\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.123726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.123712    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-model-cache\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.123919 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.123895    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tmp-dir\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.148955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.148926    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a4455f\"/\"kube-root-ca.crt\""
Nov 03 12:38:05.148955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.148930    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a4455f\"/\"default-dockercfg-9bpgn\""
Nov 03 12:38:05.149103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.149047    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a4455f\"/\"praxis-route-a4455f-kserve-self-signed-certs\""
Nov 03 12:38:05.229088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.228468    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-model-cache\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.229540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.229498    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tmp-dir\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.232097 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.230060    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-home\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.232665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.232633    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzg6v\" (UniqueName: \"kubernetes.io/projected/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-kube-api-access-fzg6v\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.233270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.233248    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tls-certs\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.236516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.236136    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-dshm\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.236516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.232563    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-home\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.236516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.229968    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tmp-dir\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.236516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.229048    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-model-cache\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.238423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.238395    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tls-certs\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.239232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.239162    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-dshm\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.286172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.286136    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzg6v\" (UniqueName: \"kubernetes.io/projected/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-kube-api-access-fzg6v\") pod \"praxis-route-a4455f-kserve-69d98cd8cf-r2djg\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") " pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.365648 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.365568    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:05.752533 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:05.752502    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"]
Nov 03 12:38:05.753486 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:38:05.753455    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ad59b72_c2e5_4815_9a87_83763ecf4fa4.slice/crio-24b338c456b0e68bf7627095488df8d7330b84557cd22fe5a591194f0ed12ab2 WatchSource:0}: Error finding container 24b338c456b0e68bf7627095488df8d7330b84557cd22fe5a591194f0ed12ab2: Status 404 returned error can't find the container with id 24b338c456b0e68bf7627095488df8d7330b84557cd22fe5a591194f0ed12ab2
Nov 03 12:38:06.261779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:06.261745    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg" event={"ID":"3ad59b72-c2e5-4815-9a87-83763ecf4fa4","Type":"ContainerStarted","Data":"a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285"}
Nov 03 12:38:06.262157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:06.261787    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg" event={"ID":"3ad59b72-c2e5-4815-9a87-83763ecf4fa4","Type":"ContainerStarted","Data":"24b338c456b0e68bf7627095488df8d7330b84557cd22fe5a591194f0ed12ab2"}
Nov 03 12:38:06.365722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:06.365672    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:06.367949 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:06.367904    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg" podStartSLOduration=1.367888474 podStartE2EDuration="1.367888474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:38:06.366292448 +0000 UTC m=+2835.378759111" watchObservedRunningTime="2026-11-03 12:38:06.367888474 +0000 UTC m=+2835.380355194"
Nov 03 12:38:09.564875 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:38:09.564841    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:38:15.365886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:15.365854    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:15.382416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:15.382388    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:16.335575 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:16.335544    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:38:39.565043 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:38:39.565007    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:38:44.126414 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.126380    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"]
Nov 03 12:38:44.126910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.126563    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy" containerID="cri-o://e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d" gracePeriod=30
Nov 03 12:38:44.172298 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.172269    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"]
Nov 03 12:38:44.172474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.172448    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy" containerID="cri-o://927effa03de3dd5ae6bb4e1558a2055853bc2fc92350f63b7a225876bd83a580" gracePeriod=30
Nov 03 12:38:44.175071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.175039    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"]
Nov 03 12:38:44.175239 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.175218    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy" containerID="cri-o://fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186" gracePeriod=30
Nov 03 12:38:44.220489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.220451    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"]
Nov 03 12:38:44.220731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:44.220688    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy" containerID="cri-o://7105956ad7fb3fd5818134f840ca6c18c956f819d55fcf59484ff6335db51626" gracePeriod=30
Nov 03 12:38:46.149595 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:46.149559    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"]
Nov 03 12:38:46.150019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:46.149807    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy" containerID="cri-o://d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0" gracePeriod=30
Nov 03 12:38:49.448388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.448355    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:38:49.542278 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542242    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-data\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.542455 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542337    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-certs\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.542455 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542395    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-credential-socket\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.542567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542478    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/629e5415-662a-4211-8ec6-a2b32497de0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629e5415-662a-4211-8ec6-a2b32497de0b-istio-podinfo\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.542618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542567    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-istio-token\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-istio-token\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.542779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542750    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-envoy\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.542933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542915    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629e5415-662a-4211-8ec6-a2b32497de0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629e5415-662a-4211-8ec6-a2b32497de0b-istiod-ca-cert\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.543010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.542969    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-kube-api-access-zgxg9\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-kube-api-access-zgxg9\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.543065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.543010    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-socket\") pod \"629e5415-662a-4211-8ec6-a2b32497de0b\" (UID: \"629e5415-662a-4211-8ec6-a2b32497de0b\") "
Nov 03 12:38:49.543862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.543837    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-socket" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.544073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.544053    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-data" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.545004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.544977    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-credential-socket" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.545127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.545035    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-certs" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.545239 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.545215    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629e5415-662a-4211-8ec6-a2b32497de0b-istiod-ca-cert" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:49.549966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.549841    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-envoy" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.549966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.549862    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629e5415-662a-4211-8ec6-a2b32497de0b-istio-podinfo" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:49.549966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.549889    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-kube-api-access-zgxg9" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "kube-api-access-zgxg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.549966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.549922    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-istio-token" pod "629e5415-662a-4211-8ec6-a2b32497de0b" (UID: "629e5415-662a-4211-8ec6-a2b32497de0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.553953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.553931    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:38:49.557320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.557296    4909 generic.go:356] "Generic (PLEG): container finished" podID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerID="fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186" exitCode=0
Nov 03 12:38:49.557492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.557343    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"
Nov 03 12:38:49.557492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.557389    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" event={"ID":"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92","Type":"ContainerDied","Data":"fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186"}
Nov 03 12:38:49.557492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.557451    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl" event={"ID":"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92","Type":"ContainerDied","Data":"636024e9146b121e937a1a242611c4df3020f2e5333a430d7fc2dedda6db2f65"}
Nov 03 12:38:49.557492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.557481    4909 scope.go:122] "RemoveContainer" containerID="fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186"
Nov 03 12:38:49.559336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.559305    4909 generic.go:356] "Generic (PLEG): container finished" podID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerID="927effa03de3dd5ae6bb4e1558a2055853bc2fc92350f63b7a225876bd83a580" exitCode=0
Nov 03 12:38:49.559446 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.559370    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" event={"ID":"f40ca4ba-5907-469a-9b72-86e2cf143699","Type":"ContainerDied","Data":"927effa03de3dd5ae6bb4e1558a2055853bc2fc92350f63b7a225876bd83a580"}
Nov 03 12:38:49.561092 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.561061    4909 generic.go:356] "Generic (PLEG): container finished" podID="629e5415-662a-4211-8ec6-a2b32497de0b" containerID="e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d" exitCode=0
Nov 03 12:38:49.561196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.561131    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" event={"ID":"629e5415-662a-4211-8ec6-a2b32497de0b","Type":"ContainerDied","Data":"e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d"}
Nov 03 12:38:49.561196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.561170    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc" event={"ID":"629e5415-662a-4211-8ec6-a2b32497de0b","Type":"ContainerDied","Data":"18f3ca1c79720e474c07b7ec5d8345e95991cc2cfc9346c6a9caf6df12277838"}
Nov 03 12:38:49.561316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.561242    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"
Nov 03 12:38:49.566959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.566941    4909 generic.go:356] "Generic (PLEG): container finished" podID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerID="7105956ad7fb3fd5818134f840ca6c18c956f819d55fcf59484ff6335db51626" exitCode=0
Nov 03 12:38:49.567065 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.566981    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" event={"ID":"ec2fdd5a-fb58-4278-89bc-3f79da765f56","Type":"ContainerDied","Data":"7105956ad7fb3fd5818134f840ca6c18c956f819d55fcf59484ff6335db51626"}
Nov 03 12:38:49.569208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.569189    4909 scope.go:122] "RemoveContainer" containerID="fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186"
Nov 03 12:38:49.569469 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:38:49.569450    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186\": container with ID starting with fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186 not found: ID does not exist" containerID="fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186"
Nov 03 12:38:49.569550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.569479    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186"} err="failed to get container status \"fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186\": rpc error: code = NotFound desc = could not find container \"fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186\": container with ID starting with fbb31a3e342da0951efa11f45a687a2190b4041ecd3496e944150293a2506186 not found: ID does not exist"
Nov 03 12:38:49.569550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.569496    4909 scope.go:122] "RemoveContainer" containerID="e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d"
Nov 03 12:38:49.583574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.583556    4909 scope.go:122] "RemoveContainer" containerID="e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d"
Nov 03 12:38:49.584687 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:38:49.584661    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d\": container with ID starting with e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d not found: ID does not exist" containerID="e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d"
Nov 03 12:38:49.584808 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.584692    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d"} err="failed to get container status \"e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d\": rpc error: code = NotFound desc = could not find container \"e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d\": container with ID starting with e8b84f56d65eaccce8faff5f6767cab889f02e9fe408bcd8335a2854b273187d not found: ID does not exist"
Nov 03 12:38:49.584808 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.584732    4909 scope.go:122] "RemoveContainer" containerID="009f55c1050061fecfb26d7e183dc3b8424e026afe3f0e6497a222e2b3dc7cfc"
Nov 03 12:38:49.592273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.592255    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:38:49.609640 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.608523    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:38:49.644993 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.644959    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-podinfo\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.645159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645014    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-credential-socket\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.645159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645060    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-token\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-token\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.645159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645103    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-socket\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.645339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645164    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-envoy\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.645339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645208    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istiod-ca-cert\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.645339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645268    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-certs\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.645339 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645314    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-data\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.645556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645350    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-credential-socket\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.645556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645347    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-credential-socket" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.645556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645402    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-socket\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.645556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645473    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-kube-api-access-98sts\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-kube-api-access-98sts\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.645556 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645518    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-envoy\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.645853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645578    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-socket" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.645853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645586    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-kube-api-access-v2fdt\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-kube-api-access-v2fdt\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.645853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645619    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-token\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-token\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.645853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645654    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-data\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.645853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645681    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40ca4ba-5907-469a-9b72-86e2cf143699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40ca4ba-5907-469a-9b72-86e2cf143699-istiod-ca-cert\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.645853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645774    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-token\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-token\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.645853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645837    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-certs\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645884    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-podinfo\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645913    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istiod-ca-cert" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645930    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-podinfo\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.645961    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-envoy\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646014    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-credential-socket\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646072    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-socket\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646103    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-certs\") pod \"f40ca4ba-5907-469a-9b72-86e2cf143699\" (UID: \"f40ca4ba-5907-469a-9b72-86e2cf143699\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646132    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-kube-api-access-jjvxz\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-kube-api-access-jjvxz\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.646235 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646195    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-data\") pod \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\" (UID: \"ec2fdd5a-fb58-4278-89bc-3f79da765f56\") "
Nov 03 12:38:49.646732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646240    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istiod-ca-cert\") pod \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\" (UID: \"e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92\") "
Nov 03 12:38:49.646732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646657    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-certs" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.646925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.646902    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-data" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.647024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647001    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istiod-ca-cert" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647378    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-certs" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647487    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647512    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647528    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647543    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647559    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647573    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629e5415-662a-4211-8ec6-a2b32497de0b-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647587    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647601    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647615    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647629    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647643    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647658    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629e5415-662a-4211-8ec6-a2b32497de0b-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647673    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgxg9\" (UniqueName: \"kubernetes.io/projected/629e5415-662a-4211-8ec6-a2b32497de0b-kube-api-access-zgxg9\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647688    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.647858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647719    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629e5415-662a-4211-8ec6-a2b32497de0b-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.648540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.647958    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-podinfo" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:49.648540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.648187    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-credential-socket" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.648540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.648102    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40ca4ba-5907-469a-9b72-86e2cf143699-istiod-ca-cert" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:49.648540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.648371    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-socket" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.648540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.648391    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-data" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.648982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.648959    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-data" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.649407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.649146    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-credential-socket" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.649407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.649211    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-certs" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.649407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.649328    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-socket" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.650572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.650543    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-podinfo" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:49.651027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.651002    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-kube-api-access-jjvxz" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "kube-api-access-jjvxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.651171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.651117    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-envoy" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.652006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.651981    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-podinfo" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:49.653232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.653173    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-envoy" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.653232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.653190    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-envoy" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.653232 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.653189    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-token" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.653406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.653260    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-token" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.653446 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.653423    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-kube-api-access-v2fdt" pod "ec2fdd5a-fb58-4278-89bc-3f79da765f56" (UID: "ec2fdd5a-fb58-4278-89bc-3f79da765f56"). InnerVolumeSpecName "kube-api-access-v2fdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.653483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.653468    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-token" pod "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" (UID: "e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.655156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.655133    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-kube-api-access-98sts" pod "f40ca4ba-5907-469a-9b72-86e2cf143699" (UID: "f40ca4ba-5907-469a-9b72-86e2cf143699"). InnerVolumeSpecName "kube-api-access-98sts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749071    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749103    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749116    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749129    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749141    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749193    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvxz\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-kube-api-access-jjvxz\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749218    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749231    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749244    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749257    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749271    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749286    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749298    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40ca4ba-5907-469a-9b72-86e2cf143699-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749309    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98sts\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-kube-api-access-98sts\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749321    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749330    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2fdt\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-kube-api-access-v2fdt\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749341    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2fdd5a-fb58-4278-89bc-3f79da765f56-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749352    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40ca4ba-5907-469a-9b72-86e2cf143699-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749362    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40ca4ba-5907-469a-9b72-86e2cf143699-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749376    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2fdd5a-fb58-4278-89bc-3f79da765f56-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.750737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:49.749389    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:50.046875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.046798    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"]
Nov 03 12:38:50.053198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.053170    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-7j9vc"]
Nov 03 12:38:50.157757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.157726    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"]
Nov 03 12:38:50.164632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.164609    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-6lqtl"]
Nov 03 12:38:50.246120 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.246093    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:38:50.574318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.574290    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"
Nov 03 12:38:50.574761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.574322    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh" event={"ID":"ec2fdd5a-fb58-4278-89bc-3f79da765f56","Type":"ContainerDied","Data":"6b8acb75e66771c1707ad78f5f01594050afa45fb688a4caadfea7ef6cea07b1"}
Nov 03 12:38:50.574761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.574374    4909 scope.go:122] "RemoveContainer" containerID="7105956ad7fb3fd5818134f840ca6c18c956f819d55fcf59484ff6335db51626"
Nov 03 12:38:50.577377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.577359    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"
Nov 03 12:38:50.577488 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.577405    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh" event={"ID":"f40ca4ba-5907-469a-9b72-86e2cf143699","Type":"ContainerDied","Data":"b8e70d8bf55a19ea471a35b38dc0b54e57879bd3b7f8edcc2cf44a61118724d2"}
Nov 03 12:38:50.585044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.585027    4909 scope.go:122] "RemoveContainer" containerID="927effa03de3dd5ae6bb4e1558a2055853bc2fc92350f63b7a225876bd83a580"
Nov 03 12:38:50.726460 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.726425    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"]
Nov 03 12:38:50.731304 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.731269    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-n97vh"]
Nov 03 12:38:50.833860 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.833782    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"]
Nov 03 12:38:50.836863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:50.836839    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-824nh"]
Nov 03 12:38:51.135362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.135282    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" path="/var/lib/kubelet/pods/629e5415-662a-4211-8ec6-a2b32497de0b/volumes"
Nov 03 12:38:51.135946 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.135927    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" path="/var/lib/kubelet/pods/e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92/volumes"
Nov 03 12:38:51.136515 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.136501    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" path="/var/lib/kubelet/pods/ec2fdd5a-fb58-4278-89bc-3f79da765f56/volumes"
Nov 03 12:38:51.137516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.137499    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" path="/var/lib/kubelet/pods/f40ca4ba-5907-469a-9b72-86e2cf143699/volumes"
Nov 03 12:38:51.415492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.415466    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:38:51.475788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.475760    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-credential-socket\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.475907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.475797    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-socket\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.475907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.475867    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-kube-api-access-2ztrl\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-kube-api-access-2ztrl\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.475907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.475894    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24197277-684e-4545-9b2c-1aaba25a5f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24197277-684e-4545-9b2c-1aaba25a5f3c-istiod-ca-cert\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.476034 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.475928    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-podinfo\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.476079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476038    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-credential-socket" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:51.476079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476057    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-token\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.476176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476156    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-socket" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:51.476228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476175    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-data\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.476228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476217    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-envoy\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.476312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476269    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24197277-684e-4545-9b2c-1aaba25a5f3c-istiod-ca-cert" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:51.476312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476305    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-certs\") pod \"24197277-684e-4545-9b2c-1aaba25a5f3c\" (UID: \"24197277-684e-4545-9b2c-1aaba25a5f3c\") "
Nov 03 12:38:51.476422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476371    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-data" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:51.476648 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476624    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-certs" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:51.476870 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476850    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.476870 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476868    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.477043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476879    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.477043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476887    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.477043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.476896    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24197277-684e-4545-9b2c-1aaba25a5f3c-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.478213 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.478191    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-podinfo" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:51.478316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.478277    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-envoy" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:51.478526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.478507    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-token" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:51.478567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.478526    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-kube-api-access-2ztrl" pod "24197277-684e-4545-9b2c-1aaba25a5f3c" (UID: "24197277-684e-4545-9b2c-1aaba25a5f3c"). InnerVolumeSpecName "kube-api-access-2ztrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:51.578322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.578291    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztrl\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-kube-api-access-2ztrl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.578322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.578320    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.578823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.578330    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.578823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.578339    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24197277-684e-4545-9b2c-1aaba25a5f3c-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:51.585980 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.585949    4909 generic.go:356] "Generic (PLEG): container finished" podID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerID="d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0" exitCode=0
Nov 03 12:38:51.586091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.585997    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"
Nov 03 12:38:51.586091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.586027    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" event={"ID":"24197277-684e-4545-9b2c-1aaba25a5f3c","Type":"ContainerDied","Data":"d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0"}
Nov 03 12:38:51.586091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.586074    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh" event={"ID":"24197277-684e-4545-9b2c-1aaba25a5f3c","Type":"ContainerDied","Data":"58967c485f7de6341a705df9819cf95166ab1fa4c0dac7b800179072c054a97c"}
Nov 03 12:38:51.586237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.586097    4909 scope.go:122] "RemoveContainer" containerID="d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0"
Nov 03 12:38:51.596224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.596208    4909 scope.go:122] "RemoveContainer" containerID="d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0"
Nov 03 12:38:51.596490 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:38:51.596472    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0\": container with ID starting with d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0 not found: ID does not exist" containerID="d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0"
Nov 03 12:38:51.596537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.596497    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0"} err="failed to get container status \"d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0\": rpc error: code = NotFound desc = could not find container \"d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0\": container with ID starting with d05da0e333ac2c61fcc710fdcae5e1c15d4ad09f4ac93d17141e3d20d7962ee0 not found: ID does not exist"
Nov 03 12:38:51.740050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.740014    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"]
Nov 03 12:38:51.742728 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:51.742691    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-gjkzh"]
Nov 03 12:38:53.124413 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:38:53.124379    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" path="/var/lib/kubelet/pods/24197277-684e-4545-9b2c-1aaba25a5f3c/volumes"
Nov 03 12:39:09.564812 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:39:09.564778    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:39:17.078199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078168    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"]
Nov 03 12:39:17.078756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078741    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy"
Nov 03 12:39:17.078756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078756    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078766    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078771    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078779    4909 cpu_manager.go:414] "containerMap: removing container" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078784    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078792    4909 cpu_manager.go:414] "containerMap: removing container" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078797    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078824    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy"
Nov 03 12:39:17.078853 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078829    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy"
Nov 03 12:39:17.079111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078920    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7fb80b6-94a2-47c4-a1e8-de75b1ad3e92" containerName="istio-proxy"
Nov 03 12:39:17.079111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078930    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy"
Nov 03 12:39:17.079111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078935    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="629e5415-662a-4211-8ec6-a2b32497de0b" containerName="istio-proxy"
Nov 03 12:39:17.079111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078942    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="24197277-684e-4545-9b2c-1aaba25a5f3c" containerName="istio-proxy"
Nov 03 12:39:17.079111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078953    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40ca4ba-5907-469a-9b72-86e2cf143699" containerName="istio-proxy"
Nov 03 12:39:17.079111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.078959    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy"
Nov 03 12:39:17.091554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.091528    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.143891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.143860    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"]
Nov 03 12:39:17.162578 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.162547    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tmp-dir\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.162762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.162605    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47j5h\" (UniqueName: \"kubernetes.io/projected/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-kube-api-access-47j5h\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.162762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.162642    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tls-certs\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.162876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.162773    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-home\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.162876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.162813    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-model-cache\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.163013 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.162996    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-dshm\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.179484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.179461    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-561bd4\"/\"default-dockercfg-knlx5\""
Nov 03 12:39:17.179593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.179574    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-561bd4\"/\"autores-model-561bd4-kserve-self-signed-certs\""
Nov 03 12:39:17.185573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.185554    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-561bd4\"/\"kube-root-ca.crt\""
Nov 03 12:39:17.185682 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.185604    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-561bd4\"/\"openshift-service-ca.crt\""
Nov 03 12:39:17.264024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.263992    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-dshm\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264076    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tmp-dir\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264118    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47j5h\" (UniqueName: \"kubernetes.io/projected/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-kube-api-access-47j5h\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264154    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tls-certs\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264198    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-home\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264221    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-model-cache\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264486 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264466    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tmp-dir\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264559    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-home\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.264671 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.264650    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-model-cache\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.266382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.266359    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-dshm\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.266536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.266520    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tls-certs\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.316255 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.316230    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47j5h\" (UniqueName: \"kubernetes.io/projected/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-kube-api-access-47j5h\") pod \"autores-model-561bd4-kserve-666c569589-vqgkr\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") " pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.403758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.403675    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:17.770515 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:17.770490    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"]
Nov 03 12:39:17.772187 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:39:17.772153    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbe37f76_9c2f_4cb0_a93f_25c033c472d3.slice/crio-3a63ee775b310d3bf7ffce39f11f107a1b359873472cd0d009341fde0ece0a78 WatchSource:0}: Error finding container 3a63ee775b310d3bf7ffce39f11f107a1b359873472cd0d009341fde0ece0a78: Status 404 returned error can't find the container with id 3a63ee775b310d3bf7ffce39f11f107a1b359873472cd0d009341fde0ece0a78
Nov 03 12:39:18.759759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:18.759718    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr" event={"ID":"dbe37f76-9c2f-4cb0-a93f-25c033c472d3","Type":"ContainerStarted","Data":"7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7"}
Nov 03 12:39:18.760157 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:18.759773    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr" event={"ID":"dbe37f76-9c2f-4cb0-a93f-25c033c472d3","Type":"ContainerStarted","Data":"3a63ee775b310d3bf7ffce39f11f107a1b359873472cd0d009341fde0ece0a78"}
Nov 03 12:39:18.865645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:18.865592    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr" podStartSLOduration=1.865577826 podStartE2EDuration="1.865577826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:39:18.864270497 +0000 UTC m=+2907.876737162" watchObservedRunningTime="2026-11-03 12:39:18.865577826 +0000 UTC m=+2907.878044488"
Nov 03 12:39:19.404109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:19.404079    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:27.404395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:27.404364    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:27.416466 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:27.416443    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:27.833666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:27.833639    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:39:29.602375 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:29.602344    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:39:37.803777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.803685    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"]
Nov 03 12:39:37.804313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.804299    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy"
Nov 03 12:39:37.804359 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.804315    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2fdd5a-fb58-4278-89bc-3f79da765f56" containerName="istio-proxy"
Nov 03 12:39:37.807621 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.807604    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.857370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.857340    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"openshift-service-ca.crt\""
Nov 03 12:39:37.896774 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.896738    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92w4h\" (UniqueName: \"kubernetes.io/projected/3ff7d855-087e-4353-b807-c831d74489fc-kube-api-access-92w4h\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.896929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.896802    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7d855-087e-4353-b807-c831d74489fc-tls-certs\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.896929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.896908    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-tmp-dir\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.897043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.896948    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-home\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.897136 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.897114    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-model-cache\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.897198 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.897168    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-dshm\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.903101 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.903083    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"default-dockercfg-htfq5\""
Nov 03 12:39:37.903195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.903083    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"kube-root-ca.crt\""
Nov 03 12:39:37.903195 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.903175    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"auth-test-model-e0615c-kserve-self-signed-certs\""
Nov 03 12:39:37.906393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.906371    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"]
Nov 03 12:39:37.998164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998131    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-model-cache\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998174    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-dshm\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998241    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92w4h\" (UniqueName: \"kubernetes.io/projected/3ff7d855-087e-4353-b807-c831d74489fc-kube-api-access-92w4h\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998270    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7d855-087e-4353-b807-c831d74489fc-tls-certs\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998327    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-tmp-dir\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998444    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-home\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998575    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-model-cache\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998743 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998693    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-tmp-dir\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:37.998816 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:37.998757    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-home\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:38.000522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:38.000502    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-dshm\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:38.000811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:38.000794    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7d855-087e-4353-b807-c831d74489fc-tls-certs\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:38.053057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:38.053030    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92w4h\" (UniqueName: \"kubernetes.io/projected/3ff7d855-087e-4353-b807-c831d74489fc-kube-api-access-92w4h\") pod \"auth-test-model-e0615c-kserve-6777569ff7-cl7j2\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:38.121322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:38.121264    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:38.488527 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:38.488500    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"]
Nov 03 12:39:38.489762 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:39:38.489738    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ff7d855_087e_4353_b807_c831d74489fc.slice/crio-7b3bdb728bd8182caaa3eb9c56e9991ae7667c23d936d4f6b071179e627bd91a WatchSource:0}: Error finding container 7b3bdb728bd8182caaa3eb9c56e9991ae7667c23d936d4f6b071179e627bd91a: Status 404 returned error can't find the container with id 7b3bdb728bd8182caaa3eb9c56e9991ae7667c23d936d4f6b071179e627bd91a
Nov 03 12:39:38.895057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:38.894969    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2" event={"ID":"3ff7d855-087e-4353-b807-c831d74489fc","Type":"ContainerStarted","Data":"036020e869dbfa13c709a42e6f667d135eb3afac737f6507f79b60e1f18b41c1"}
Nov 03 12:39:38.895057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:38.895010    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2" event={"ID":"3ff7d855-087e-4353-b807-c831d74489fc","Type":"ContainerStarted","Data":"7b3bdb728bd8182caaa3eb9c56e9991ae7667c23d936d4f6b071179e627bd91a"}
Nov 03 12:39:39.004260 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:39.004202    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2" podStartSLOduration=2.004189631 podStartE2EDuration="2.004189631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:39:39.000451104 +0000 UTC m=+2928.012917788" watchObservedRunningTime="2026-11-03 12:39:39.004189631 +0000 UTC m=+2928.016656294"
Nov 03 12:39:39.133828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:39.133588    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:39.564978 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:39:39.564942    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:39:48.122073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:48.122033    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:48.134543 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:48.134520    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:39:48.974008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:39:48.973981    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:40:09.564846 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:40:09.564813    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:40:14.206313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:14.206277    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"]
Nov 03 12:40:16.201837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:16.201804    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"]
Nov 03 12:40:27.748037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:27.748005    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:40:29.226843 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.226807    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr" podUID="dbe37f76-9c2f-4cb0-a93f-25c033c472d3" containerName="main" containerID="cri-o://7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7" gracePeriod=45
Nov 03 12:40:29.501548 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.501520    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:40:29.605181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605149    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-dshm\") pod \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") "
Nov 03 12:40:29.605316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605295    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-kube-api-access-47j5h\" (UniqueName: \"kubernetes.io/projected/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-kube-api-access-47j5h\") pod \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") "
Nov 03 12:40:29.605359 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605340    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tmp-dir\") pod \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") "
Nov 03 12:40:29.605403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605366    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-home\") pod \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") "
Nov 03 12:40:29.605490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605471    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-model-cache\") pod \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") "
Nov 03 12:40:29.605590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605568    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tls-certs\") pod \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\" (UID: \"dbe37f76-9c2f-4cb0-a93f-25c033c472d3\") "
Nov 03 12:40:29.605590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605578    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tmp-dir" pod "dbe37f76-9c2f-4cb0-a93f-25c033c472d3" (UID: "dbe37f76-9c2f-4cb0-a93f-25c033c472d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:29.605807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605594    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-home" pod "dbe37f76-9c2f-4cb0-a93f-25c033c472d3" (UID: "dbe37f76-9c2f-4cb0-a93f-25c033c472d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:29.605871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.605811    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-model-cache" pod "dbe37f76-9c2f-4cb0-a93f-25c033c472d3" (UID: "dbe37f76-9c2f-4cb0-a93f-25c033c472d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:29.606141 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.606120    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:29.606205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.606146    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:29.606205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.606163    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:29.607537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.607517    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-dshm" pod "dbe37f76-9c2f-4cb0-a93f-25c033c472d3" (UID: "dbe37f76-9c2f-4cb0-a93f-25c033c472d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:29.607626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.607588    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tls-certs" pod "dbe37f76-9c2f-4cb0-a93f-25c033c472d3" (UID: "dbe37f76-9c2f-4cb0-a93f-25c033c472d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:40:29.607626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.607613    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-kube-api-access-47j5h" pod "dbe37f76-9c2f-4cb0-a93f-25c033c472d3" (UID: "dbe37f76-9c2f-4cb0-a93f-25c033c472d3"). InnerVolumeSpecName "kube-api-access-47j5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:40:29.707169 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.707142    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:29.707169 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.707168    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:29.707357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:29.707178    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47j5h\" (UniqueName: \"kubernetes.io/projected/dbe37f76-9c2f-4cb0-a93f-25c033c472d3-kube-api-access-47j5h\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:30.213179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.213144    4909 generic.go:356] "Generic (PLEG): container finished" podID="dbe37f76-9c2f-4cb0-a93f-25c033c472d3" containerID="7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7" exitCode=0
Nov 03 12:40:30.213179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.213181    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"
Nov 03 12:40:30.213423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.213181    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr" event={"ID":"dbe37f76-9c2f-4cb0-a93f-25c033c472d3","Type":"ContainerDied","Data":"7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7"}
Nov 03 12:40:30.213423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.213355    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr" event={"ID":"dbe37f76-9c2f-4cb0-a93f-25c033c472d3","Type":"ContainerDied","Data":"3a63ee775b310d3bf7ffce39f11f107a1b359873472cd0d009341fde0ece0a78"}
Nov 03 12:40:30.213423 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.213375    4909 scope.go:122] "RemoveContainer" containerID="7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7"
Nov 03 12:40:30.223425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.223408    4909 scope.go:122] "RemoveContainer" containerID="7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7"
Nov 03 12:40:30.223666 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:40:30.223650    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7\": container with ID starting with 7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7 not found: ID does not exist" containerID="7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7"
Nov 03 12:40:30.223729 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.223672    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7"} err="failed to get container status \"7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7\": rpc error: code = NotFound desc = could not find container \"7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7\": container with ID starting with 7185ba9f5796c63f5d1a4fad8181290145569cda47eaa78bc80a242c29b660b7 not found: ID does not exist"
Nov 03 12:40:30.365730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.365686    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"]
Nov 03 12:40:30.411718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:30.411666    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-561bd4/autores-model-561bd4-kserve-666c569589-vqgkr"]
Nov 03 12:40:31.124781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.124753    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe37f76-9c2f-4cb0-a93f-25c033c472d3" path="/var/lib/kubelet/pods/dbe37f76-9c2f-4cb0-a93f-25c033c472d3/volumes"
Nov 03 12:40:31.222686 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.222647    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg" podUID="3ad59b72-c2e5-4815-9a87-83763ecf4fa4" containerName="main" containerID="cri-o://a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285" gracePeriod=45
Nov 03 12:40:31.324563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.324529    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg" podUID="3ad59b72-c2e5-4815-9a87-83763ecf4fa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": dial tcp 10.132.0.153:8000: connect: connection refused"
Nov 03 12:40:31.496535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.496513    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:40:31.629263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.629234    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tls-certs\") pod \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") "
Nov 03 12:40:31.629469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.629273    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-model-cache\") pod \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") "
Nov 03 12:40:31.629469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.629307    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-home\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-home\") pod \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") "
Nov 03 12:40:31.629469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.629334    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-kube-api-access-fzg6v\" (UniqueName: \"kubernetes.io/projected/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-kube-api-access-fzg6v\") pod \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") "
Nov 03 12:40:31.629469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.629421    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-dshm\") pod \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") "
Nov 03 12:40:31.629469 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.629453    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tmp-dir\") pod \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\" (UID: \"3ad59b72-c2e5-4815-9a87-83763ecf4fa4\") "
Nov 03 12:40:31.630186 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.630159    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-model-cache" pod "3ad59b72-c2e5-4815-9a87-83763ecf4fa4" (UID: "3ad59b72-c2e5-4815-9a87-83763ecf4fa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:31.630353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.630291    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tmp-dir" pod "3ad59b72-c2e5-4815-9a87-83763ecf4fa4" (UID: "3ad59b72-c2e5-4815-9a87-83763ecf4fa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:31.630610 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.630593    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-home" pod "3ad59b72-c2e5-4815-9a87-83763ecf4fa4" (UID: "3ad59b72-c2e5-4815-9a87-83763ecf4fa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:31.632118 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.632089    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tls-certs" pod "3ad59b72-c2e5-4815-9a87-83763ecf4fa4" (UID: "3ad59b72-c2e5-4815-9a87-83763ecf4fa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:40:31.632344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.632316    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-kube-api-access-fzg6v" pod "3ad59b72-c2e5-4815-9a87-83763ecf4fa4" (UID: "3ad59b72-c2e5-4815-9a87-83763ecf4fa4"). InnerVolumeSpecName "kube-api-access-fzg6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:40:31.632614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.632587    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-dshm" pod "3ad59b72-c2e5-4815-9a87-83763ecf4fa4" (UID: "3ad59b72-c2e5-4815-9a87-83763ecf4fa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:31.731435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.731405    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:31.731435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.731433    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:31.731644 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.731442    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:31.731644 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.731453    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:31.731644 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.731463    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzg6v\" (UniqueName: \"kubernetes.io/projected/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-kube-api-access-fzg6v\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:31.731644 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:31.731473    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad59b72-c2e5-4815-9a87-83763ecf4fa4-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:32.228058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.228032    4909 generic.go:356] "Generic (PLEG): container finished" podID="3ad59b72-c2e5-4815-9a87-83763ecf4fa4" containerID="a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285" exitCode=0
Nov 03 12:40:32.228223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.228080    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"
Nov 03 12:40:32.228223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.228100    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg" event={"ID":"3ad59b72-c2e5-4815-9a87-83763ecf4fa4","Type":"ContainerDied","Data":"a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285"}
Nov 03 12:40:32.228223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.228130    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg" event={"ID":"3ad59b72-c2e5-4815-9a87-83763ecf4fa4","Type":"ContainerDied","Data":"24b338c456b0e68bf7627095488df8d7330b84557cd22fe5a591194f0ed12ab2"}
Nov 03 12:40:32.228223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.228145    4909 scope.go:122] "RemoveContainer" containerID="a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285"
Nov 03 12:40:32.244062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.244039    4909 scope.go:122] "RemoveContainer" containerID="a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285"
Nov 03 12:40:32.244341 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:40:32.244318    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285\": container with ID starting with a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285 not found: ID does not exist" containerID="a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285"
Nov 03 12:40:32.244389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.244349    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285"} err="failed to get container status \"a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285\": rpc error: code = NotFound desc = could not find container \"a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285\": container with ID starting with a9fb3aae16d1ae40c0922274d37b44ca548f3be2e8f8e53c210dc4488d9f7285 not found: ID does not exist"
Nov 03 12:40:32.380670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.380633    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"]
Nov 03 12:40:32.386107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:32.386079    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-a4455f/praxis-route-a4455f-kserve-69d98cd8cf-r2djg"]
Nov 03 12:40:33.123730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:33.123683    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad59b72-c2e5-4815-9a87-83763ecf4fa4" path="/var/lib/kubelet/pods/3ad59b72-c2e5-4815-9a87-83763ecf4fa4/volumes"
Nov 03 12:40:39.564104 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:40:39.564070    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:40:51.505800 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:51.505769    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:40:51.512072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:51.512045    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:40:54.512563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:54.512529    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"]
Nov 03 12:40:54.610192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:54.610160    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"]
Nov 03 12:40:56.729348 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:56.729312    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"]
Nov 03 12:40:56.826280 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:56.826246    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"]
Nov 03 12:40:59.513308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:59.513270    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" podUID="5e8cd76c-1938-4645-bd13-ac37ca22f213" containerName="payload-processing" containerID="cri-o://83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c" gracePeriod=25
Nov 03 12:40:59.610773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:40:59.610735    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" podUID="916f7473-2148-476c-a59c-1ec89776f020" containerName="payload-pre-processing" containerID="cri-o://d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383" gracePeriod=25
Nov 03 12:41:01.730388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:01.730301    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng" podUID="184adf72-7527-4b4e-a053-b691555cff90" containerName="payload-processing" containerID="cri-o://0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702" gracePeriod=25
Nov 03 12:41:01.826968 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:01.826931    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq" podUID="1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" containerName="payload-pre-processing" containerID="cri-o://d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7" gracePeriod=25
Nov 03 12:41:02.055838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.055798    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" podUID="916f7473-2148-476c-a59c-1ec89776f020" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:41:02.056401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.056366    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" podUID="5e8cd76c-1938-4645-bd13-ac37ca22f213" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:41:02.112990 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.112963    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:41:02.137250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.137229    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:41:02.217801 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.217777    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-kube-api-access-bl24s\" (UniqueName: \"kubernetes.io/projected/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-kube-api-access-bl24s\") pod \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") "
Nov 03 12:41:02.217908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.217819    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184adf72-7527-4b4e-a053-b691555cff90-kube-api-access-9tgt6\" (UniqueName: \"kubernetes.io/projected/184adf72-7527-4b4e-a053-b691555cff90-kube-api-access-9tgt6\") pod \"184adf72-7527-4b4e-a053-b691555cff90\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") "
Nov 03 12:41:02.217908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.217858    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184adf72-7527-4b4e-a053-b691555cff90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184adf72-7527-4b4e-a053-b691555cff90-plugins-config-volume\") pod \"184adf72-7527-4b4e-a053-b691555cff90\" (UID: \"184adf72-7527-4b4e-a053-b691555cff90\") "
Nov 03 12:41:02.218182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.218126    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-plugins-config-volume\") pod \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\" (UID: \"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2\") "
Nov 03 12:41:02.218943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.218374    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184adf72-7527-4b4e-a053-b691555cff90-plugins-config-volume" pod "184adf72-7527-4b4e-a053-b691555cff90" (UID: "184adf72-7527-4b4e-a053-b691555cff90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:02.218943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.218608    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-plugins-config-volume" pod "1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" (UID: "1ed6a15f-7792-4c94-83a6-c9a8803c6ca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:02.218943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.218929    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:02.218943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.218945    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184adf72-7527-4b4e-a053-b691555cff90-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:02.220594 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.220564    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184adf72-7527-4b4e-a053-b691555cff90-kube-api-access-9tgt6" pod "184adf72-7527-4b4e-a053-b691555cff90" (UID: "184adf72-7527-4b4e-a053-b691555cff90"). InnerVolumeSpecName "kube-api-access-9tgt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:02.220989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.220965    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-kube-api-access-bl24s" pod "1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" (UID: "1ed6a15f-7792-4c94-83a6-c9a8803c6ca2"). InnerVolumeSpecName "kube-api-access-bl24s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:02.319840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.319755    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl24s\" (UniqueName: \"kubernetes.io/projected/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2-kube-api-access-bl24s\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:02.319840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.319785    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgt6\" (UniqueName: \"kubernetes.io/projected/184adf72-7527-4b4e-a053-b691555cff90-kube-api-access-9tgt6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:02.416104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.416073    4909 generic.go:356] "Generic (PLEG): container finished" podID="1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" containerID="d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7" exitCode=0
Nov 03 12:41:02.416305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.416128    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"
Nov 03 12:41:02.416305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.416158    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq" event={"ID":"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2","Type":"ContainerDied","Data":"d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7"}
Nov 03 12:41:02.416305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.416224    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq" event={"ID":"1ed6a15f-7792-4c94-83a6-c9a8803c6ca2","Type":"ContainerDied","Data":"c340dad41c85e91dd1e577e5f1bc1eb1aa3e529a2356a47d1f9ed4a90744398f"}
Nov 03 12:41:02.416305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.416241    4909 scope.go:122] "RemoveContainer" containerID="d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7"
Nov 03 12:41:02.417776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.417752    4909 generic.go:356] "Generic (PLEG): container finished" podID="184adf72-7527-4b4e-a053-b691555cff90" containerID="0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702" exitCode=0
Nov 03 12:41:02.417906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.417791    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng" event={"ID":"184adf72-7527-4b4e-a053-b691555cff90","Type":"ContainerDied","Data":"0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702"}
Nov 03 12:41:02.417906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.417798    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"
Nov 03 12:41:02.417906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.417830    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng" event={"ID":"184adf72-7527-4b4e-a053-b691555cff90","Type":"ContainerDied","Data":"f6907c7f936625b7aa70b2070a0e7718cabbefdbbb8a35eafc848c9812e028e8"}
Nov 03 12:41:02.426503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.426483    4909 scope.go:122] "RemoveContainer" containerID="d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7"
Nov 03 12:41:02.426762 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:02.426738    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7\": container with ID starting with d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7 not found: ID does not exist" containerID="d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7"
Nov 03 12:41:02.426857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.426768    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7"} err="failed to get container status \"d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7\": rpc error: code = NotFound desc = could not find container \"d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7\": container with ID starting with d31daac3bf60c10091872a06c972ce2394bbb6df0f63343df895ec062200a4a7 not found: ID does not exist"
Nov 03 12:41:02.426857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.426786    4909 scope.go:122] "RemoveContainer" containerID="0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702"
Nov 03 12:41:02.435504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.435479    4909 scope.go:122] "RemoveContainer" containerID="0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702"
Nov 03 12:41:02.435845 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:02.435826    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702\": container with ID starting with 0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702 not found: ID does not exist" containerID="0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702"
Nov 03 12:41:02.435920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.435851    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702"} err="failed to get container status \"0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702\": rpc error: code = NotFound desc = could not find container \"0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702\": container with ID starting with 0dbe198140be2cb9a69fb76748a6319b18ca8e4fa8c1b029ab55d1946210b702 not found: ID does not exist"
Nov 03 12:41:02.566995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.566960    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"]
Nov 03 12:41:02.573370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.573309    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-561bd4-6b94655d7c-tkqng"]
Nov 03 12:41:02.672904 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.672872    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"]
Nov 03 12:41:02.675761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:02.675735    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-561bd4-5cb4494475-s8jfq"]
Nov 03 12:41:03.124886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:03.124848    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184adf72-7527-4b4e-a053-b691555cff90" path="/var/lib/kubelet/pods/184adf72-7527-4b4e-a053-b691555cff90/volumes"
Nov 03 12:41:03.125294 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:03.125280    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" path="/var/lib/kubelet/pods/1ed6a15f-7792-4c94-83a6-c9a8803c6ca2/volumes"
Nov 03 12:41:09.564209 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:09.564175    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:41:12.056080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:12.056035    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" podUID="5e8cd76c-1938-4645-bd13-ac37ca22f213" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:41:12.056460 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:12.056039    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" podUID="916f7473-2148-476c-a59c-1ec89776f020" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:41:15.404997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.404966    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"]
Nov 03 12:41:15.405514 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.405176    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk" podUID="70caac4b-48cc-48af-b03e-5d5c4e943421" containerName="maas-api" containerID="cri-o://bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e" gracePeriod=30
Nov 03 12:41:15.707584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.707557    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:41:15.870779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.870733    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls\") pod \"70caac4b-48cc-48af-b03e-5d5c4e943421\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") "
Nov 03 12:41:15.870779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.870791    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-metrics-certs\") pod \"70caac4b-48cc-48af-b03e-5d5c4e943421\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") "
Nov 03 12:41:15.871052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.870818    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70caac4b-48cc-48af-b03e-5d5c4e943421-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70caac4b-48cc-48af-b03e-5d5c4e943421-openshift-service-ca\") pod \"70caac4b-48cc-48af-b03e-5d5c4e943421\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") "
Nov 03 12:41:15.871052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.870944    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70caac4b-48cc-48af-b03e-5d5c4e943421-kube-api-access-xnv5v\" (UniqueName: \"kubernetes.io/projected/70caac4b-48cc-48af-b03e-5d5c4e943421-kube-api-access-xnv5v\") pod \"70caac4b-48cc-48af-b03e-5d5c4e943421\" (UID: \"70caac4b-48cc-48af-b03e-5d5c4e943421\") "
Nov 03 12:41:15.871254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.871228    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70caac4b-48cc-48af-b03e-5d5c4e943421-openshift-service-ca" pod "70caac4b-48cc-48af-b03e-5d5c4e943421" (UID: "70caac4b-48cc-48af-b03e-5d5c4e943421"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:15.871378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.871365    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70caac4b-48cc-48af-b03e-5d5c4e943421-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:15.873142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.873111    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls" pod "70caac4b-48cc-48af-b03e-5d5c4e943421" (UID: "70caac4b-48cc-48af-b03e-5d5c4e943421"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:41:15.873251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.873178    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70caac4b-48cc-48af-b03e-5d5c4e943421-kube-api-access-xnv5v" pod "70caac4b-48cc-48af-b03e-5d5c4e943421" (UID: "70caac4b-48cc-48af-b03e-5d5c4e943421"). InnerVolumeSpecName "kube-api-access-xnv5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:15.873251 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.873217    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-metrics-certs" pod "70caac4b-48cc-48af-b03e-5d5c4e943421" (UID: "70caac4b-48cc-48af-b03e-5d5c4e943421"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:41:15.972353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.972318    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:15.972353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.972347    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70caac4b-48cc-48af-b03e-5d5c4e943421-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:15.972353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:15.972359    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnv5v\" (UniqueName: \"kubernetes.io/projected/70caac4b-48cc-48af-b03e-5d5c4e943421-kube-api-access-xnv5v\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:16.506473 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.506439    4909 generic.go:356] "Generic (PLEG): container finished" podID="70caac4b-48cc-48af-b03e-5d5c4e943421" containerID="bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e" exitCode=0
Nov 03 12:41:16.506936 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.506488    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"
Nov 03 12:41:16.506936 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.506519    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk" event={"ID":"70caac4b-48cc-48af-b03e-5d5c4e943421","Type":"ContainerDied","Data":"bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e"}
Nov 03 12:41:16.506936 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.506577    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk" event={"ID":"70caac4b-48cc-48af-b03e-5d5c4e943421","Type":"ContainerDied","Data":"467b0ada4069571f329ae5806f19f2e7428a43d49118576912ef7fbd5e194c54"}
Nov 03 12:41:16.506936 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.506599    4909 scope.go:122] "RemoveContainer" containerID="bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e"
Nov 03 12:41:16.517307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.517292    4909 scope.go:122] "RemoveContainer" containerID="bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e"
Nov 03 12:41:16.517578 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:16.517560    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e\": container with ID starting with bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e not found: ID does not exist" containerID="bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e"
Nov 03 12:41:16.517643 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.517586    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e"} err="failed to get container status \"bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e\": rpc error: code = NotFound desc = could not find container \"bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e\": container with ID starting with bb83091d91241d0c3e183edf06143f6ee587003b3cbbdf588ccd1d610f32f74e not found: ID does not exist"
Nov 03 12:41:16.658689 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.658658    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"]
Nov 03 12:41:16.661299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:16.661275    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4455f-8fc9958fc-pqzmk"]
Nov 03 12:41:17.122979 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:17.122948    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70caac4b-48cc-48af-b03e-5d5c4e943421" path="/var/lib/kubelet/pods/70caac4b-48cc-48af-b03e-5d5c4e943421/volumes"
Nov 03 12:41:19.800812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.800788    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:41:19.813677 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.813644    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8cd76c-1938-4645-bd13-ac37ca22f213-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8cd76c-1938-4645-bd13-ac37ca22f213-plugins-config-volume\") pod \"5e8cd76c-1938-4645-bd13-ac37ca22f213\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") "
Nov 03 12:41:19.813975 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.813735    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8cd76c-1938-4645-bd13-ac37ca22f213-kube-api-access-6fm46\" (UniqueName: \"kubernetes.io/projected/5e8cd76c-1938-4645-bd13-ac37ca22f213-kube-api-access-6fm46\") pod \"5e8cd76c-1938-4645-bd13-ac37ca22f213\" (UID: \"5e8cd76c-1938-4645-bd13-ac37ca22f213\") "
Nov 03 12:41:19.814103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.814066    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8cd76c-1938-4645-bd13-ac37ca22f213-plugins-config-volume" pod "5e8cd76c-1938-4645-bd13-ac37ca22f213" (UID: "5e8cd76c-1938-4645-bd13-ac37ca22f213"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:19.814321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.814303    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8cd76c-1938-4645-bd13-ac37ca22f213-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:19.821943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.821914    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8cd76c-1938-4645-bd13-ac37ca22f213-kube-api-access-6fm46" pod "5e8cd76c-1938-4645-bd13-ac37ca22f213" (UID: "5e8cd76c-1938-4645-bd13-ac37ca22f213"). InnerVolumeSpecName "kube-api-access-6fm46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:19.881932 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.881905    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:41:19.914939 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.914910    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916f7473-2148-476c-a59c-1ec89776f020-kube-api-access-2pf6l\" (UniqueName: \"kubernetes.io/projected/916f7473-2148-476c-a59c-1ec89776f020-kube-api-access-2pf6l\") pod \"916f7473-2148-476c-a59c-1ec89776f020\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") "
Nov 03 12:41:19.915092 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.914956    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916f7473-2148-476c-a59c-1ec89776f020-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f7473-2148-476c-a59c-1ec89776f020-plugins-config-volume\") pod \"916f7473-2148-476c-a59c-1ec89776f020\" (UID: \"916f7473-2148-476c-a59c-1ec89776f020\") "
Nov 03 12:41:19.915406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.915378    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916f7473-2148-476c-a59c-1ec89776f020-plugins-config-volume" pod "916f7473-2148-476c-a59c-1ec89776f020" (UID: "916f7473-2148-476c-a59c-1ec89776f020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:19.915521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.915415    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fm46\" (UniqueName: \"kubernetes.io/projected/5e8cd76c-1938-4645-bd13-ac37ca22f213-kube-api-access-6fm46\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:19.917355 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:19.917321    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916f7473-2148-476c-a59c-1ec89776f020-kube-api-access-2pf6l" pod "916f7473-2148-476c-a59c-1ec89776f020" (UID: "916f7473-2148-476c-a59c-1ec89776f020"). InnerVolumeSpecName "kube-api-access-2pf6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:20.016109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.016073    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pf6l\" (UniqueName: \"kubernetes.io/projected/916f7473-2148-476c-a59c-1ec89776f020-kube-api-access-2pf6l\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:20.016109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.016104    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f7473-2148-476c-a59c-1ec89776f020-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:20.417116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.417083    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:41:20.537949 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.537918    4909 generic.go:356] "Generic (PLEG): container finished" podID="916f7473-2148-476c-a59c-1ec89776f020" containerID="d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383" exitCode=0
Nov 03 12:41:20.538094 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.537981    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"
Nov 03 12:41:20.538094 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.537975    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" event={"ID":"916f7473-2148-476c-a59c-1ec89776f020","Type":"ContainerDied","Data":"d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383"}
Nov 03 12:41:20.538206 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.538114    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8" event={"ID":"916f7473-2148-476c-a59c-1ec89776f020","Type":"ContainerDied","Data":"eb335f9add0618b624f4576599db5a667a53cb70fcb3937ea4c25163cf032134"}
Nov 03 12:41:20.538206 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.538129    4909 scope.go:122] "RemoveContainer" containerID="d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383"
Nov 03 12:41:20.539720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.539683    4909 generic.go:356] "Generic (PLEG): container finished" podID="5e8cd76c-1938-4645-bd13-ac37ca22f213" containerID="83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c" exitCode=0
Nov 03 12:41:20.539937 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.539748    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"
Nov 03 12:41:20.540050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.539745    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" event={"ID":"5e8cd76c-1938-4645-bd13-ac37ca22f213","Type":"ContainerDied","Data":"83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c"}
Nov 03 12:41:20.540050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.540040    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7" event={"ID":"5e8cd76c-1938-4645-bd13-ac37ca22f213","Type":"ContainerDied","Data":"3714080a841e31873bed4b203311125d3b96307443c3d1230ef735082d7eb614"}
Nov 03 12:41:20.548010 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.547989    4909 scope.go:122] "RemoveContainer" containerID="d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383"
Nov 03 12:41:20.548492 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:20.548470    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383\": container with ID starting with d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383 not found: ID does not exist" containerID="d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383"
Nov 03 12:41:20.548590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.548501    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383"} err="failed to get container status \"d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383\": rpc error: code = NotFound desc = could not find container \"d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383\": container with ID starting with d791710d60d45b6d827d02db7dbc32cc52267ebfb9cd2c47101d196be677b383 not found: ID does not exist"
Nov 03 12:41:20.548590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.548522    4909 scope.go:122] "RemoveContainer" containerID="83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c"
Nov 03 12:41:20.557596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.557578    4909 scope.go:122] "RemoveContainer" containerID="83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c"
Nov 03 12:41:20.557850 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:20.557830    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c\": container with ID starting with 83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c not found: ID does not exist" containerID="83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c"
Nov 03 12:41:20.557912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.557855    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c"} err="failed to get container status \"83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c\": rpc error: code = NotFound desc = could not find container \"83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c\": container with ID starting with 83f49f42aba78ef676e172de61b5136b2afc447f2d79e990f0cf40bcfc34e35c not found: ID does not exist"
Nov 03 12:41:20.708742 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.708712    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"]
Nov 03 12:41:20.714249 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.714226    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4455f-76fcd696ff-gx8x8"]
Nov 03 12:41:20.817929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.817901    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"]
Nov 03 12:41:20.824144 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:20.824118    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4455f-767f54d6c7-k8zc7"]
Nov 03 12:41:21.124110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:21.124033    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8cd76c-1938-4645-bd13-ac37ca22f213" path="/var/lib/kubelet/pods/5e8cd76c-1938-4645-bd13-ac37ca22f213/volumes"
Nov 03 12:41:21.124471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:21.124458    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916f7473-2148-476c-a59c-1ec89776f020" path="/var/lib/kubelet/pods/916f7473-2148-476c-a59c-1ec89776f020/volumes"
Nov 03 12:41:32.297912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:32.297879    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"]
Nov 03 12:41:32.298412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:32.298073    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" containerID="cri-o://3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091" gracePeriod=30
Nov 03 12:41:33.095607 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:33.095574    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"]
Nov 03 12:41:33.196729 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:33.196687    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"]
Nov 03 12:41:36.093110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:36.093078    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"]
Nov 03 12:41:36.093593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:36.093290    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy" containerID="cri-o://a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d" gracePeriod=30
Nov 03 12:41:37.227397 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.227360    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"]
Nov 03 12:41:37.228149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228125    4909 cpu_manager.go:414] "containerMap: removing container" podUID="184adf72-7527-4b4e-a053-b691555cff90" containerName="payload-processing"
Nov 03 12:41:37.228149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228150    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184adf72-7527-4b4e-a053-b691555cff90" containerName="payload-processing"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228177    4909 cpu_manager.go:414] "containerMap: removing container" podUID="dbe37f76-9c2f-4cb0-a93f-25c033c472d3" containerName="main"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228185    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe37f76-9c2f-4cb0-a93f-25c033c472d3" containerName="main"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228214    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3ad59b72-c2e5-4815-9a87-83763ecf4fa4" containerName="main"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228222    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad59b72-c2e5-4815-9a87-83763ecf4fa4" containerName="main"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228237    4909 cpu_manager.go:414] "containerMap: removing container" podUID="70caac4b-48cc-48af-b03e-5d5c4e943421" containerName="maas-api"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228244    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70caac4b-48cc-48af-b03e-5d5c4e943421" containerName="maas-api"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228260    4909 cpu_manager.go:414] "containerMap: removing container" podUID="916f7473-2148-476c-a59c-1ec89776f020" containerName="payload-pre-processing"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228267    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916f7473-2148-476c-a59c-1ec89776f020" containerName="payload-pre-processing"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228282    4909 cpu_manager.go:414] "containerMap: removing container" podUID="1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" containerName="payload-pre-processing"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228289    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" containerName="payload-pre-processing"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228304    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5e8cd76c-1938-4645-bd13-ac37ca22f213" containerName="payload-processing"
Nov 03 12:41:37.228328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228311    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8cd76c-1938-4645-bd13-ac37ca22f213" containerName="payload-processing"
Nov 03 12:41:37.228770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228451    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad59b72-c2e5-4815-9a87-83763ecf4fa4" containerName="main"
Nov 03 12:41:37.228770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228462    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="70caac4b-48cc-48af-b03e-5d5c4e943421" containerName="maas-api"
Nov 03 12:41:37.228770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228471    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="916f7473-2148-476c-a59c-1ec89776f020" containerName="payload-pre-processing"
Nov 03 12:41:37.228770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228481    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed6a15f-7792-4c94-83a6-c9a8803c6ca2" containerName="payload-pre-processing"
Nov 03 12:41:37.228770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228491    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8cd76c-1938-4645-bd13-ac37ca22f213" containerName="payload-processing"
Nov 03 12:41:37.228770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228506    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="184adf72-7527-4b4e-a053-b691555cff90" containerName="payload-processing"
Nov 03 12:41:37.228770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.228518    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe37f76-9c2f-4cb0-a93f-25c033c472d3" containerName="main"
Nov 03 12:41:37.233122 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.233104    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.292004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.291969    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"]
Nov 03 12:41:37.328633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.328605    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-52ab15-openshift-default-dockercfg-dxwhd\""
Nov 03 12:41:37.378425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.378096    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:41:37.378425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.378168    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 12:41:37.389809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.389771    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-data\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.389982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.389893    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjrdt\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-kube-api-access-fjrdt\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.389982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.389951    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-certs\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.390104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.390053    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-envoy\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.390169 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.390146    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-credential-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.390231 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.390183    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.390287 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.390254    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-istio-token\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.390345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.390300    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04708f3d-162a-4077-af9d-ab6d62b12040-istio-podinfo\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.390395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.390365    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04708f3d-162a-4077-af9d-ab6d62b12040-istiod-ca-cert\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.491930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.491534    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrdt\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-kube-api-access-fjrdt\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.491930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.491593    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-certs\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.491930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.491631    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-envoy\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.491930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.491681    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-credential-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.492243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.491957    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.492243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.492065    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-istio-token\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.492243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.492101    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04708f3d-162a-4077-af9d-ab6d62b12040-istio-podinfo\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.492243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.492171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04708f3d-162a-4077-af9d-ab6d62b12040-istiod-ca-cert\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.492243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.492207    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-data\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.493017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.492993    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-credential-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.493221 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.493203    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.493413 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.493395    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-data\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.494089 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.494037    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-certs\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.494215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.494196    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04708f3d-162a-4077-af9d-ab6d62b12040-istiod-ca-cert\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.495978 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.495955    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-envoy\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.496409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.496391    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04708f3d-162a-4077-af9d-ab6d62b12040-istio-podinfo\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.552208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.552176    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-istio-token\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.552555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.552532    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrdt\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-kube-api-access-fjrdt\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:37.606291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.606262    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:41:37.693994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.693959    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-data\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694042    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istiod-ca-cert\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694146    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-certs\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694207    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-token\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-token\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694206    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-data" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:37.694324 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694282    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-envoy\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694327    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-socket\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694359    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-podinfo\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694382    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-certs" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:37.694495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694432    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-credential-socket\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694445    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istiod-ca-cert" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:37.694495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694474    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-kube-api-access-whqsw\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-kube-api-access-whqsw\") pod \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\" (UID: \"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d\") "
Nov 03 12:41:37.694847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694634    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-credential-socket" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:37.694847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.694658    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-socket" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:37.695043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.695024    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.695108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.695048    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.695108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.695062    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.695108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.695077    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.695108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.695090    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.696911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.696885    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-podinfo" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:41:37.696911 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.696889    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-envoy" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:37.697187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.697168    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-kube-api-access-whqsw" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "kube-api-access-whqsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:37.697256 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.697195    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-token" pod "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" (UID: "bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:37.707316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.707291    4909 generic.go:356] "Generic (PLEG): container finished" podID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerID="3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091" exitCode=0
Nov 03 12:41:37.707409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.707353    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"
Nov 03 12:41:37.707409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.707374    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" event={"ID":"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d","Type":"ContainerDied","Data":"3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091"}
Nov 03 12:41:37.707520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.707428    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc" event={"ID":"bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d","Type":"ContainerDied","Data":"e92c8f6141c2861f87dd93c181396b0d10d98cedfd9f1516a1009ac9c38c6bd0"}
Nov 03 12:41:37.707520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.707452    4909 scope.go:122] "RemoveContainer" containerID="3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091"
Nov 03 12:41:37.719258 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.719235    4909 scope.go:122] "RemoveContainer" containerID="3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091"
Nov 03 12:41:37.719608 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:37.719579    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091\": container with ID starting with 3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091 not found: ID does not exist" containerID="3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091"
Nov 03 12:41:37.719748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.719611    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091"} err="failed to get container status \"3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091\": rpc error: code = NotFound desc = could not find container \"3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091\": container with ID starting with 3f20d722096770608842b7839423e65b6c2df4aaac8e0dd683b660f61cd35091 not found: ID does not exist"
Nov 03 12:41:37.796395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.796316    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.796395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.796345    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.796395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.796360    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqsw\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-kube-api-access-whqsw\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.796395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.796374    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:37.821637 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.821608    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"]
Nov 03 12:41:37.826184 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.826153    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-dt5nc"]
Nov 03 12:41:37.846403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:37.846376    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:38.096780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.096649    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp" podUID="11b73485-b056-4b49-991f-9be46681cba8" containerName="payload-processing" containerID="cri-o://324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b" gracePeriod=25
Nov 03 12:41:38.197027 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.196990    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh" podUID="282ce652-1641-4658-a189-eeaff2904167" containerName="payload-pre-processing" containerID="cri-o://1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba" gracePeriod=25
Nov 03 12:41:38.262762 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.262735    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"]
Nov 03 12:41:38.265066 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:41:38.265040    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04708f3d_162a_4077_af9d_ab6d62b12040.slice/crio-c71216be0f800d78591910c3c2e986c3549a69bdb7dfb65a159bf6c5e5c22317 WatchSource:0}: Error finding container c71216be0f800d78591910c3c2e986c3549a69bdb7dfb65a159bf6c5e5c22317: Status 404 returned error can't find the container with id c71216be0f800d78591910c3c2e986c3549a69bdb7dfb65a159bf6c5e5c22317
Nov 03 12:41:38.302881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.302847    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:41:38.302984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.302924    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:41:38.302984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.302952    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:41:38.394798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.394775    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:41:38.490372 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.490348    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:41:38.505128 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.505099    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b73485-b056-4b49-991f-9be46681cba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b73485-b056-4b49-991f-9be46681cba8-plugins-config-volume\") pod \"11b73485-b056-4b49-991f-9be46681cba8\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") "
Nov 03 12:41:38.505275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.505166    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b73485-b056-4b49-991f-9be46681cba8-kube-api-access-j4cjs\" (UniqueName: \"kubernetes.io/projected/11b73485-b056-4b49-991f-9be46681cba8-kube-api-access-j4cjs\") pod \"11b73485-b056-4b49-991f-9be46681cba8\" (UID: \"11b73485-b056-4b49-991f-9be46681cba8\") "
Nov 03 12:41:38.505722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.505670    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b73485-b056-4b49-991f-9be46681cba8-plugins-config-volume" pod "11b73485-b056-4b49-991f-9be46681cba8" (UID: "11b73485-b056-4b49-991f-9be46681cba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:38.513572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.508457    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b73485-b056-4b49-991f-9be46681cba8-kube-api-access-j4cjs" pod "11b73485-b056-4b49-991f-9be46681cba8" (UID: "11b73485-b056-4b49-991f-9be46681cba8"). InnerVolumeSpecName "kube-api-access-j4cjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:38.613135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.613049    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282ce652-1641-4658-a189-eeaff2904167-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282ce652-1641-4658-a189-eeaff2904167-plugins-config-volume\") pod \"282ce652-1641-4658-a189-eeaff2904167\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") "
Nov 03 12:41:38.613306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.613187    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282ce652-1641-4658-a189-eeaff2904167-kube-api-access-m295t\" (UniqueName: \"kubernetes.io/projected/282ce652-1641-4658-a189-eeaff2904167-kube-api-access-m295t\") pod \"282ce652-1641-4658-a189-eeaff2904167\" (UID: \"282ce652-1641-4658-a189-eeaff2904167\") "
Nov 03 12:41:38.613966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.613937    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b73485-b056-4b49-991f-9be46681cba8-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:38.613966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.613967    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4cjs\" (UniqueName: \"kubernetes.io/projected/11b73485-b056-4b49-991f-9be46681cba8-kube-api-access-j4cjs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:38.616502 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.616439    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282ce652-1641-4658-a189-eeaff2904167-kube-api-access-m295t" pod "282ce652-1641-4658-a189-eeaff2904167" (UID: "282ce652-1641-4658-a189-eeaff2904167"). InnerVolumeSpecName "kube-api-access-m295t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:38.622209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.616843    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282ce652-1641-4658-a189-eeaff2904167-plugins-config-volume" pod "282ce652-1641-4658-a189-eeaff2904167" (UID: "282ce652-1641-4658-a189-eeaff2904167"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:38.715163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.715128    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282ce652-1641-4658-a189-eeaff2904167-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:38.715163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.715161    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m295t\" (UniqueName: \"kubernetes.io/projected/282ce652-1641-4658-a189-eeaff2904167-kube-api-access-m295t\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:38.717441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.717417    4909 generic.go:356] "Generic (PLEG): container finished" podID="282ce652-1641-4658-a189-eeaff2904167" containerID="1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba" exitCode=0
Nov 03 12:41:38.717561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.717490    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"
Nov 03 12:41:38.717561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.717504    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh" event={"ID":"282ce652-1641-4658-a189-eeaff2904167","Type":"ContainerDied","Data":"1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba"}
Nov 03 12:41:38.717714 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.717593    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh" event={"ID":"282ce652-1641-4658-a189-eeaff2904167","Type":"ContainerDied","Data":"aad5999b2d7394cb6d472a5eee688eedf160804ca8692bd5025e41a0f9a4fca3"}
Nov 03 12:41:38.717714 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.717623    4909 scope.go:122] "RemoveContainer" containerID="1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba"
Nov 03 12:41:38.718995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.718975    4909 generic.go:356] "Generic (PLEG): container finished" podID="11b73485-b056-4b49-991f-9be46681cba8" containerID="324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b" exitCode=0
Nov 03 12:41:38.719088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.719054    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp" event={"ID":"11b73485-b056-4b49-991f-9be46681cba8","Type":"ContainerDied","Data":"324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b"}
Nov 03 12:41:38.719156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.719094    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"
Nov 03 12:41:38.719156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.719121    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp" event={"ID":"11b73485-b056-4b49-991f-9be46681cba8","Type":"ContainerDied","Data":"016676204fd84b7a0a92d01c43db21c0e00f5710d15d2f296225b4ab95e21e54"}
Nov 03 12:41:38.722109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.722088    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" event={"ID":"04708f3d-162a-4077-af9d-ab6d62b12040","Type":"ContainerStarted","Data":"efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834"}
Nov 03 12:41:38.722203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.722124    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" event={"ID":"04708f3d-162a-4077-af9d-ab6d62b12040","Type":"ContainerStarted","Data":"c71216be0f800d78591910c3c2e986c3549a69bdb7dfb65a159bf6c5e5c22317"}
Nov 03 12:41:38.729865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.728405    4909 scope.go:122] "RemoveContainer" containerID="1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba"
Nov 03 12:41:38.729865 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:38.728803    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba\": container with ID starting with 1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba not found: ID does not exist" containerID="1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba"
Nov 03 12:41:38.729865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.728837    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba"} err="failed to get container status \"1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba\": rpc error: code = NotFound desc = could not find container \"1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba\": container with ID starting with 1ef44cb514e267b132d0b9b4c02caaaed55b0625de46d9b787bf1486193446ba not found: ID does not exist"
Nov 03 12:41:38.729865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.728860    4909 scope.go:122] "RemoveContainer" containerID="324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b"
Nov 03 12:41:38.739040 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.739009    4909 scope.go:122] "RemoveContainer" containerID="324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b"
Nov 03 12:41:38.739336 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:38.739313    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b\": container with ID starting with 324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b not found: ID does not exist" containerID="324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b"
Nov 03 12:41:38.739404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.739342    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b"} err="failed to get container status \"324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b\": rpc error: code = NotFound desc = could not find container \"324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b\": container with ID starting with 324bd829737ebfc73e2584d9af0ed1429f92c40b9cb41c7ed6d6f4029bd6338b not found: ID does not exist"
Nov 03 12:41:38.847341 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.847309    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:38.877033 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.876948    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podStartSLOduration=1.8769345720000001 podStartE2EDuration="1.876934572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:41:38.875218337 +0000 UTC m=+3047.887684999" watchObservedRunningTime="2026-11-03 12:41:38.876934572 +0000 UTC m=+3047.889401234"
Nov 03 12:41:38.978525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.978494    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"]
Nov 03 12:41:38.981202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:38.981180    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3176d9-59979c4b69-gxmqp"]
Nov 03 12:41:39.083983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:39.083951    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"]
Nov 03 12:41:39.087812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:39.087782    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3176d9-76484d959-wbkxh"]
Nov 03 12:41:39.126037 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:39.126010    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b73485-b056-4b49-991f-9be46681cba8" path="/var/lib/kubelet/pods/11b73485-b056-4b49-991f-9be46681cba8/volumes"
Nov 03 12:41:39.126517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:39.126495    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282ce652-1641-4658-a189-eeaff2904167" path="/var/lib/kubelet/pods/282ce652-1641-4658-a189-eeaff2904167/volumes"
Nov 03 12:41:39.127008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:39.126989    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" path="/var/lib/kubelet/pods/bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d/volumes"
Nov 03 12:41:39.565053 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:39.565024    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:41:39.852867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:39.852839    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:40.737765 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:40.737735    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:40.738520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:40.738501    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:41:41.380305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.380283    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:41:41.553342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553311    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-envoy\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553371    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-podinfo\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553436    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-certs\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553485 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553469    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-kube-api-access-mj6f6\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-kube-api-access-mj6f6\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553495    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-credential-socket\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553555    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-data\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553631 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553585    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-socket\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553669    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istiod-ca-cert\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553737    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-token\") pod \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\" (UID: \"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d\") "
Nov 03 12:41:41.553841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553821    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-certs" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:41.554006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553886    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-credential-socket" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:41.554006 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.553938    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-socket" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:41.554166 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.554140    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istiod-ca-cert" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:41.554259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.554236    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-data" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:41.554476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.554458    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.554539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.554484    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.554539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.554501    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.554539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.554517    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.554539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.554535    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.555635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.555612    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-envoy" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:41.555909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.555885    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-token" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:41.556012 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.555984    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-kube-api-access-mj6f6" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "kube-api-access-mj6f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:41.556060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.556035    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-podinfo" pod "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" (UID: "2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:41:41.655468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.655390    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.655468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.655415    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.655468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.655428    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj6f6\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-kube-api-access-mj6f6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.655468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.655438    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:41.745344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.745314    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerID="a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d" exitCode=0
Nov 03 12:41:41.745854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.745362    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"
Nov 03 12:41:41.745854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.745393    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" event={"ID":"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d","Type":"ContainerDied","Data":"a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d"}
Nov 03 12:41:41.745854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.745448    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx" event={"ID":"2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d","Type":"ContainerDied","Data":"29fac1dcead3ebd58b49a5ee26b086d0c5c4405e626f9f42251c89cc0ef9a7a2"}
Nov 03 12:41:41.745854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.745474    4909 scope.go:122] "RemoveContainer" containerID="a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d"
Nov 03 12:41:41.758667 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.757260    4909 scope.go:122] "RemoveContainer" containerID="643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21"
Nov 03 12:41:41.774582 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.774563    4909 scope.go:122] "RemoveContainer" containerID="a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d"
Nov 03 12:41:41.774885 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:41.774865    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d\": container with ID starting with a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d not found: ID does not exist" containerID="a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d"
Nov 03 12:41:41.774958 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.774891    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d"} err="failed to get container status \"a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d\": rpc error: code = NotFound desc = could not find container \"a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d\": container with ID starting with a8161c0b52eb538c71fec6462ae52495592656b0d0d9f07a725d43e44d0f8a8d not found: ID does not exist"
Nov 03 12:41:41.774958 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.774908    4909 scope.go:122] "RemoveContainer" containerID="643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21"
Nov 03 12:41:41.775141 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:41:41.775121    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21\": container with ID starting with 643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21 not found: ID does not exist" containerID="643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21"
Nov 03 12:41:41.775182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.775146    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21"} err="failed to get container status \"643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21\": rpc error: code = NotFound desc = could not find container \"643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21\": container with ID starting with 643092ddcfaa66081ddf191e7347b88d5132b64c17e27df7371b133c0562dd21 not found: ID does not exist"
Nov 03 12:41:41.917876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.917809    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"]
Nov 03 12:41:41.928213 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:41.928190    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-561bd4-openshift-default-747678d87f-nn4lx"]
Nov 03 12:41:43.124283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:43.124249    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" path="/var/lib/kubelet/pods/2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d/volumes"
Nov 03 12:41:45.017134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017097    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"]
Nov 03 12:41:45.017879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017854    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy"
Nov 03 12:41:45.017879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017879    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy"
Nov 03 12:41:45.018056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017910    4909 cpu_manager.go:414] "containerMap: removing container" podUID="11b73485-b056-4b49-991f-9be46681cba8" containerName="payload-processing"
Nov 03 12:41:45.018056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017918    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b73485-b056-4b49-991f-9be46681cba8" containerName="payload-processing"
Nov 03 12:41:45.018056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017952    4909 cpu_manager.go:414] "containerMap: removing container" podUID="282ce652-1641-4658-a189-eeaff2904167" containerName="payload-pre-processing"
Nov 03 12:41:45.018056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017959    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282ce652-1641-4658-a189-eeaff2904167" containerName="payload-pre-processing"
Nov 03 12:41:45.018056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017972    4909 cpu_manager.go:414] "containerMap: removing container" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy"
Nov 03 12:41:45.018056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.017978    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy"
Nov 03 12:41:45.018358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.018140    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b73485-b056-4b49-991f-9be46681cba8" containerName="payload-processing"
Nov 03 12:41:45.018358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.018160    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="282ce652-1641-4658-a189-eeaff2904167" containerName="payload-pre-processing"
Nov 03 12:41:45.018358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.018171    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaafbcb-e21f-4e13-8c6f-ecb83fb12a1d" containerName="istio-proxy"
Nov 03 12:41:45.018358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.018183    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy"
Nov 03 12:41:45.018358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.018197    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy"
Nov 03 12:41:45.023417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.023395    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.072047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.072025    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-52ab15\""
Nov 03 12:41:45.093820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.093792    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72efce80-b100-4822-ad58-4cb7f99e626e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.093988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.093968    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtn7\" (UniqueName: \"kubernetes.io/projected/72efce80-b100-4822-ad58-4cb7f99e626e-kube-api-access-sxtn7\") pod \"payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.144190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.144155    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-52ab15-dockercfg-pvsq4\""
Nov 03 12:41:45.152862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.152833    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"]
Nov 03 12:41:45.193223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.193194    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"]
Nov 03 12:41:45.193768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.193752    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy"
Nov 03 12:41:45.193827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.193770    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4e83f6-33a6-411e-87e9-a31bb4ab0b6d" containerName="istio-proxy"
Nov 03 12:41:45.195087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.195005    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtn7\" (UniqueName: \"kubernetes.io/projected/72efce80-b100-4822-ad58-4cb7f99e626e-kube-api-access-sxtn7\") pod \"payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.196379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.195254    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72efce80-b100-4822-ad58-4cb7f99e626e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.196379 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.196337    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72efce80-b100-4822-ad58-4cb7f99e626e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.198322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.198007    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.237521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.237497    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"]
Nov 03 12:41:45.247955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.247926    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtn7\" (UniqueName: \"kubernetes.io/projected/72efce80-b100-4822-ad58-4cb7f99e626e-kube-api-access-sxtn7\") pod \"payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.296216 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.296148    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcrq\" (UniqueName: \"kubernetes.io/projected/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-kube-api-access-4lcrq\") pod \"payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") " pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.296216 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.296188    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") " pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.336709 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.336673    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.398064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.397653    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcrq\" (UniqueName: \"kubernetes.io/projected/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-kube-api-access-4lcrq\") pod \"payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") " pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.398064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.397743    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") " pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.399719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.399256    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") " pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.450209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.450179    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcrq\" (UniqueName: \"kubernetes.io/projected/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-kube-api-access-4lcrq\") pod \"payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") " pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.512014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.511983    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:45.698854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.698820    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"]
Nov 03 12:41:45.700794 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:41:45.700763    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72efce80_b100_4822_ad58_4cb7f99e626e.slice/crio-abf36d0fbfa85bf530fdf007972987ad6b1bc4f7069a19fae7b3ec99698d1ccd WatchSource:0}: Error finding container abf36d0fbfa85bf530fdf007972987ad6b1bc4f7069a19fae7b3ec99698d1ccd: Status 404 returned error can't find the container with id abf36d0fbfa85bf530fdf007972987ad6b1bc4f7069a19fae7b3ec99698d1ccd
Nov 03 12:41:45.804871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.804840    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g" event={"ID":"72efce80-b100-4822-ad58-4cb7f99e626e","Type":"ContainerStarted","Data":"c169af9fbfe6dfc2e8c85148c5813b24dc574bb6e0773e5be973d426a1ae5c7c"}
Nov 03 12:41:45.804992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.804906    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:45.804992 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.804933    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g" event={"ID":"72efce80-b100-4822-ad58-4cb7f99e626e","Type":"ContainerStarted","Data":"abf36d0fbfa85bf530fdf007972987ad6b1bc4f7069a19fae7b3ec99698d1ccd"}
Nov 03 12:41:45.877583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.877561    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"]
Nov 03 12:41:45.878432 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:41:45.878407    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c80eaf_c0dc_4bbf_87c5_67b1faa5ffc9.slice/crio-91629b1a3384b726f016a2dc9555b31c8808319a25aa432257d991f34e92ffe7 WatchSource:0}: Error finding container 91629b1a3384b726f016a2dc9555b31c8808319a25aa432257d991f34e92ffe7: Status 404 returned error can't find the container with id 91629b1a3384b726f016a2dc9555b31c8808319a25aa432257d991f34e92ffe7
Nov 03 12:41:45.912472 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:45.912426    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g" podStartSLOduration=1.912413972 podStartE2EDuration="1.912413972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:41:45.908387878 +0000 UTC m=+3054.920854541" watchObservedRunningTime="2026-11-03 12:41:45.912413972 +0000 UTC m=+3054.924880636"
Nov 03 12:41:46.812462 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:46.812423    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr" event={"ID":"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9","Type":"ContainerStarted","Data":"92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f"}
Nov 03 12:41:46.812894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:46.812644    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:41:46.812894 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:46.812668    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr" event={"ID":"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9","Type":"ContainerStarted","Data":"91629b1a3384b726f016a2dc9555b31c8808319a25aa432257d991f34e92ffe7"}
Nov 03 12:41:46.917706 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:46.917650    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr" podStartSLOduration=1.91763745 podStartE2EDuration="1.91763745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:41:46.91586434 +0000 UTC m=+3055.928331002" watchObservedRunningTime="2026-11-03 12:41:46.91763745 +0000 UTC m=+3055.930104112"
Nov 03 12:41:56.815499 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:56.815472    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:41:57.821792 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:41:57.821766    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:42:09.564352 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:42:09.564320    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:42:16.970368 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:16.970333    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:42:38.676189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.676113    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"]
Nov 03 12:42:38.680409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.680386    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719194    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-socket\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719338    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-credential-socket\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719473 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719395    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-envoy\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719470    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-certs\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719550    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-token\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719580    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-podinfo\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719616    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istiod-ca-cert\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719655    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-data\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.719725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.719686    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9skb\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-kube-api-access-j9skb\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.754558 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.754524    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"]
Nov 03 12:42:38.783581 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.783558    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a4489c-openshift-default-dockercfg-wxcc5\""
Nov 03 12:42:38.821290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821257    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-envoy\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821337    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-certs\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821425 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821413    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-token\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821439    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-podinfo\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821479    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istiod-ca-cert\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821552 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821515    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-data\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821547    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9skb\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-kube-api-access-j9skb\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821601    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-socket\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821727    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-credential-socket\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.821913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821794    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-certs\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.822016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.821993    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-data\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.822057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.822037    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-socket\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.822110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.822092    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-credential-socket\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.822311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.822293    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istiod-ca-cert\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.823580 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.823561    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-envoy\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.824123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.824107    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-podinfo\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.873995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.873963    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-token\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.874098 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.873995    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9skb\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-kube-api-access-j9skb\") pod \"e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") " pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:38.995746 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:38.995684    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:39.450986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:39.450958    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"]
Nov 03 12:42:39.491971 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:42:39.491936    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0e5bfe_5ef5_4b68_9734_9b15c6c9e629.slice/crio-08a81c7b5ccfd0619649222f517c206f8fb3eccac1bbdd2adb0378a155c8165b WatchSource:0}: Error finding container 08a81c7b5ccfd0619649222f517c206f8fb3eccac1bbdd2adb0378a155c8165b: Status 404 returned error can't find the container with id 08a81c7b5ccfd0619649222f517c206f8fb3eccac1bbdd2adb0378a155c8165b
Nov 03 12:42:39.494720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:39.494664    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:42:39.494812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:39.494762    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:42:39.494812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:39.494793    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:42:39.564384 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:42:39.564353    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:42:40.130871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:40.130835    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" event={"ID":"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629","Type":"ContainerStarted","Data":"2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c"}
Nov 03 12:42:40.131291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:40.130894    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" event={"ID":"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629","Type":"ContainerStarted","Data":"08a81c7b5ccfd0619649222f517c206f8fb3eccac1bbdd2adb0378a155c8165b"}
Nov 03 12:42:40.285420 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:40.285373    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" podStartSLOduration=2.285356825 podStartE2EDuration="2.285356825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:42:40.28501521 +0000 UTC m=+3109.297481873" watchObservedRunningTime="2026-11-03 12:42:40.285356825 +0000 UTC m=+3109.297823489"
Nov 03 12:42:40.996408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:40.996377    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:41.002192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:41.002167    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:41.136949 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:41.136913    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:42:41.137883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:42:41.137863    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:43:09.564832 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:43:09.564800    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:43:24.206075 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:24.206027    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:43:35.459481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:35.459450    4909 generic.go:356] "Generic (PLEG): container finished" podID="04708f3d-162a-4077-af9d-ab6d62b12040" containerID="efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834" exitCode=0
Nov 03 12:43:35.459931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:35.459529    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" event={"ID":"04708f3d-162a-4077-af9d-ab6d62b12040","Type":"ContainerDied","Data":"efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834"}
Nov 03 12:43:35.459977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:35.459939    4909 scope.go:122] "RemoveContainer" containerID="efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834"
Nov 03 12:43:35.461419 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:35.461390    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:43:35.461496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:35.461458    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:43:35.461496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:35.461490    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:43:36.470543 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:36.470511    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" event={"ID":"04708f3d-162a-4077-af9d-ab6d62b12040","Type":"ContainerStarted","Data":"871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d"}
Nov 03 12:43:36.858775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:36.847864    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:43:36.858775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:36.847986    4909 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 12:43:36.858775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:36.848025    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 12:43:37.857720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:37.854023    4909 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 12:43:37.857720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:37.854079    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 12:43:38.859722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:38.851006    4909 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 12:43:38.859722 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:38.851066    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 12:43:39.588085 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:43:39.588053    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:43:39.857773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:39.856055    4909 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 12:43:39.857773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:39.856121    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 12:43:40.749044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:40.749009    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:43:40.857730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:40.857023    4909 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 12:43:40.857730 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:40.857081    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 12:43:41.852238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:41.852205    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:43:41.853297 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:43:41.853273    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:44:06.306100 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:06.306027    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"]
Nov 03 12:44:09.564852 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:44:09.564816    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:44:21.326210 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.326174    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj" podUID="a2d08d98-334b-4bc2-bb65-f4e566bada67" containerName="main" containerID="cri-o://5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2" gracePeriod=45
Nov 03 12:44:21.590093 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.590038    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:44:21.605897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.605869    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-tmp-dir\") pod \"a2d08d98-334b-4bc2-bb65-f4e566bada67\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") "
Nov 03 12:44:21.606017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.605928    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-home\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-home\") pod \"a2d08d98-334b-4bc2-bb65-f4e566bada67\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") "
Nov 03 12:44:21.606017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.605985    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-dshm\") pod \"a2d08d98-334b-4bc2-bb65-f4e566bada67\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") "
Nov 03 12:44:21.606112 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606052    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d08d98-334b-4bc2-bb65-f4e566bada67-kube-api-access-stx9b\" (UniqueName: \"kubernetes.io/projected/a2d08d98-334b-4bc2-bb65-f4e566bada67-kube-api-access-stx9b\") pod \"a2d08d98-334b-4bc2-bb65-f4e566bada67\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") "
Nov 03 12:44:21.606179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606130    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-model-cache\") pod \"a2d08d98-334b-4bc2-bb65-f4e566bada67\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") "
Nov 03 12:44:21.606179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606161    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-tmp-dir" pod "a2d08d98-334b-4bc2-bb65-f4e566bada67" (UID: "a2d08d98-334b-4bc2-bb65-f4e566bada67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:21.606179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606173    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d08d98-334b-4bc2-bb65-f4e566bada67-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d08d98-334b-4bc2-bb65-f4e566bada67-tls-certs\") pod \"a2d08d98-334b-4bc2-bb65-f4e566bada67\" (UID: \"a2d08d98-334b-4bc2-bb65-f4e566bada67\") "
Nov 03 12:44:21.606333 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606192    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-home" pod "a2d08d98-334b-4bc2-bb65-f4e566bada67" (UID: "a2d08d98-334b-4bc2-bb65-f4e566bada67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:21.606396 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606366    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-model-cache" pod "a2d08d98-334b-4bc2-bb65-f4e566bada67" (UID: "a2d08d98-334b-4bc2-bb65-f4e566bada67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:21.606646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606628    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:21.606773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606649    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:21.606773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.606661    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:21.608317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.608293    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-dshm" pod "a2d08d98-334b-4bc2-bb65-f4e566bada67" (UID: "a2d08d98-334b-4bc2-bb65-f4e566bada67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:21.608429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.608388    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d08d98-334b-4bc2-bb65-f4e566bada67-tls-certs" pod "a2d08d98-334b-4bc2-bb65-f4e566bada67" (UID: "a2d08d98-334b-4bc2-bb65-f4e566bada67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:44:21.608484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.608442    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d08d98-334b-4bc2-bb65-f4e566bada67-kube-api-access-stx9b" pod "a2d08d98-334b-4bc2-bb65-f4e566bada67" (UID: "a2d08d98-334b-4bc2-bb65-f4e566bada67"). InnerVolumeSpecName "kube-api-access-stx9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:44:21.707494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.707466    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d08d98-334b-4bc2-bb65-f4e566bada67-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:21.707494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.707493    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stx9b\" (UniqueName: \"kubernetes.io/projected/a2d08d98-334b-4bc2-bb65-f4e566bada67-kube-api-access-stx9b\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:21.707670 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.707503    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d08d98-334b-4bc2-bb65-f4e566bada67-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:21.744473 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.744445    4909 generic.go:356] "Generic (PLEG): container finished" podID="a2d08d98-334b-4bc2-bb65-f4e566bada67" containerID="5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2" exitCode=0
Nov 03 12:44:21.744619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.744488    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"
Nov 03 12:44:21.744619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.744538    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj" event={"ID":"a2d08d98-334b-4bc2-bb65-f4e566bada67","Type":"ContainerDied","Data":"5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2"}
Nov 03 12:44:21.744619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.744579    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj" event={"ID":"a2d08d98-334b-4bc2-bb65-f4e566bada67","Type":"ContainerDied","Data":"5dfd490b45ebc7d3882a3e069cee994b909ba03e417d05c332a1e45376733e0e"}
Nov 03 12:44:21.744619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.744600    4909 scope.go:122] "RemoveContainer" containerID="5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2"
Nov 03 12:44:21.756876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.756854    4909 scope.go:122] "RemoveContainer" containerID="5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2"
Nov 03 12:44:21.757153 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:44:21.757133    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2\": container with ID starting with 5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2 not found: ID does not exist" containerID="5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2"
Nov 03 12:44:21.757228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.757161    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2"} err="failed to get container status \"5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2\": rpc error: code = NotFound desc = could not find container \"5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2\": container with ID starting with 5e8f9a0871284cc9dbba8d2904ff6e1b9fc07043fac8d876700d606ca7eb44a2 not found: ID does not exist"
Nov 03 12:44:21.896750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.896670    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"]
Nov 03 12:44:21.900111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:21.900087    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj"]
Nov 03 12:44:22.531676 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:22.531637    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-1250b3/auth-test-model-1250b3-kserve-7866ccc67c-5npxj" podUID="a2d08d98-334b-4bc2-bb65-f4e566bada67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.152:8000/ready\": context deadline exceeded"
Nov 03 12:44:23.123067 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:23.123033    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d08d98-334b-4bc2-bb65-f4e566bada67" path="/var/lib/kubelet/pods/a2d08d98-334b-4bc2-bb65-f4e566bada67/volumes"
Nov 03 12:44:25.056422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.056391    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"]
Nov 03 12:44:25.056949 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.056935    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a2d08d98-334b-4bc2-bb65-f4e566bada67" containerName="main"
Nov 03 12:44:25.057019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.056950    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d08d98-334b-4bc2-bb65-f4e566bada67" containerName="main"
Nov 03 12:44:25.057064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.057055    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d08d98-334b-4bc2-bb65-f4e566bada67" containerName="main"
Nov 03 12:44:25.061897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.061879    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.166334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.166305    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bebc6d-openshift-default-dockercfg-qr4w5\""
Nov 03 12:44:25.170416 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.170384    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"]
Nov 03 12:44:25.265793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.265753    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-podinfo\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.265982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.265812    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-credential-socket\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.265982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.265975    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-certs\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.266121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.266006    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-token\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.266121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.266067    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5sk4\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-kube-api-access-n5sk4\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.266247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.266202    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-data\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.266296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.266273    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istiod-ca-cert\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.266371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.266349    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-socket\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.266422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.266410    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-envoy\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.367953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.367883    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-certs\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.367953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.367922    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-token\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.367953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.367943    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5sk4\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-kube-api-access-n5sk4\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.367991    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-data\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368009    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istiod-ca-cert\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368040    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-socket\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368100    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-envoy\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368135    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-podinfo\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368236 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-credential-socket\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368314    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-certs\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368373    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-data\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368439    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-socket\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368608    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-credential-socket\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.368775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.368756    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istiod-ca-cert\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.370465 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.370441    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-envoy\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.370677 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.370659    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-podinfo\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.421603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.421558    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-token\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.421748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.421732    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5sk4\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-kube-api-access-n5sk4\") pod \"e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") " pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:25.674624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:25.674548    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:26.099729 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:26.099682    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"]
Nov 03 12:44:26.104461 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:26.104370    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:44:26.104578 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:26.104460    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:44:26.104578 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:26.104510    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:44:26.779715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:26.779666    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" event={"ID":"a01fbd75-25d5-4efd-8788-44a86c84cf1f","Type":"ContainerStarted","Data":"705fe81b0fb806238ac92568283ac90ba5006f9a202d1a61161383daa5b61f89"}
Nov 03 12:44:26.779920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:26.779752    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" event={"ID":"a01fbd75-25d5-4efd-8788-44a86c84cf1f","Type":"ContainerStarted","Data":"d34d5cbbd7d46149a04c25281ac762dce23fa2009376f2fde11f59b27f175377"}
Nov 03 12:44:26.933809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:26.933756    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podStartSLOduration=1.933740436 podStartE2EDuration="1.933740436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:26.933241053 +0000 UTC m=+3215.945707715" watchObservedRunningTime="2026-11-03 12:44:26.933740436 +0000 UTC m=+3215.946207100"
Nov 03 12:44:27.675070 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:27.675033    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:27.681810 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:27.681777    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:27.786460 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:27.786425    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:27.787512 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:27.787493    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:44:29.528286 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:29.528254    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:44:39.566159 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:44:39.566125    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:44:40.866706 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:40.866676    4909 generic.go:356] "Generic (PLEG): container finished" podID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerID="2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c" exitCode=0
Nov 03 12:44:40.867237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:40.866755    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" event={"ID":"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629","Type":"ContainerDied","Data":"2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c"}
Nov 03 12:44:40.867237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:40.867216    4909 scope.go:122] "RemoveContainer" containerID="2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c"
Nov 03 12:44:40.868671 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:40.868640    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:44:40.868786 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:40.868739    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:44:40.868857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:40.868786    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:44:41.137389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:41.137308    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:44:41.874138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:41.874101    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" event={"ID":"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629","Type":"ContainerStarted","Data":"5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e"}
Nov 03 12:44:41.996058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:41.996026    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:44:41.997317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:41.997296    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 12:44:41.997377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:41.997355    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 12:44:42.996418 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:42.996382    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 12:44:42.996928 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:42.996447    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 12:44:43.996754 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:43.996693    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 12:44:43.997150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:43.996769    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 12:44:44.996619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:44.996573    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 12:44:44.996823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:44.996630    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 12:44:45.996898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:45.996866    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 12:44:45.997309 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:45.996920    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 12:44:47.052273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:47.052236    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:44:47.052834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:47.052812    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:44:47.054030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:47.054009    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 12:44:47.786727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:47.786680    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"]
Nov 03 12:44:52.793216 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.793184    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"]
Nov 03 12:44:52.797030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.797008    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:52.846858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.846831    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-fd7881\""
Nov 03 12:44:52.874596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.874565    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlc6\" (UniqueName: \"kubernetes.io/projected/f1c830fa-1dcf-48a4-adbe-927013af88bb-kube-api-access-mxlc6\") pod \"payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:52.874758 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.874736    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c830fa-1dcf-48a4-adbe-927013af88bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:52.892749 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.892725    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-fd7881-dockercfg-8pzbt\""
Nov 03 12:44:52.895620 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.895597    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"]
Nov 03 12:44:52.975431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.975402    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlc6\" (UniqueName: \"kubernetes.io/projected/f1c830fa-1dcf-48a4-adbe-927013af88bb-kube-api-access-mxlc6\") pod \"payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:52.975537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.975506    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c830fa-1dcf-48a4-adbe-927013af88bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:52.976259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:52.976237    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c830fa-1dcf-48a4-adbe-927013af88bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:53.027780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:53.027757    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlc6\" (UniqueName: \"kubernetes.io/projected/f1c830fa-1dcf-48a4-adbe-927013af88bb-kube-api-access-mxlc6\") pod \"payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:53.109042 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:53.108979    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:53.471871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:53.471844    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"]
Nov 03 12:44:53.472841 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:44:53.472812    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c830fa_1dcf_48a4_adbe_927013af88bb.slice/crio-06bdf1915ba875d4cd1f721ad7aac21c19e82a929a53f5a7bac552318047b894 WatchSource:0}: Error finding container 06bdf1915ba875d4cd1f721ad7aac21c19e82a929a53f5a7bac552318047b894: Status 404 returned error can't find the container with id 06bdf1915ba875d4cd1f721ad7aac21c19e82a929a53f5a7bac552318047b894
Nov 03 12:44:53.970464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:53.970435    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8" event={"ID":"f1c830fa-1dcf-48a4-adbe-927013af88bb","Type":"ContainerStarted","Data":"8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee"}
Nov 03 12:44:53.970970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:53.970485    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8" event={"ID":"f1c830fa-1dcf-48a4-adbe-927013af88bb","Type":"ContainerStarted","Data":"06bdf1915ba875d4cd1f721ad7aac21c19e82a929a53f5a7bac552318047b894"}
Nov 03 12:44:53.970970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:53.970507    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:44:54.075767 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:44:54.075687    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8" podStartSLOduration=2.075669684 podStartE2EDuration="2.075669684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:54.074575298 +0000 UTC m=+3243.087041962" watchObservedRunningTime="2026-11-03 12:44:54.075669684 +0000 UTC m=+3243.088136350"
Nov 03 12:45:00.179573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.179545    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"]
Nov 03 12:45:00.183005 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.182985    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"]
Nov 03 12:45:00.183139 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.183095    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.186188 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.186168    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.201317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.201295    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"]
Nov 03 12:45:00.204204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.204189    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.225544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.225516    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"]
Nov 03 12:45:00.228415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.228399    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"]
Nov 03 12:45:00.228541 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.228522    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.231370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.231352    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.258642 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.258621    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ee7b15-2d11-41c8-bf55-c25468cadde7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.258738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.258677    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shzd\" (UniqueName: \"kubernetes.io/projected/57ee7b15-2d11-41c8-bf55-c25468cadde7-kube-api-access-8shzd\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.258783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.258756    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f0239a-d786-4bad-a01d-9cb637ffdcd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.258823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.258795    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvnp\" (UniqueName: \"kubernetes.io/projected/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-kube-api-access-brvnp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.258873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.258824    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86254b03-179d-466c-9ecf-2151c563f8f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.258931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.258905    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdsn\" (UniqueName: \"kubernetes.io/projected/86254b03-179d-466c-9ecf-2151c563f8f9-kube-api-access-4cdsn\") pod \"maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.259036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.259019    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqx5\" (UniqueName: \"kubernetes.io/projected/805422ad-0e4d-4e4b-a625-f622206a878c-kube-api-access-vwqx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.259081 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.259066    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/805422ad-0e4d-4e4b-a625-f622206a878c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.259130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.259093    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2sl\" (UniqueName: \"kubernetes.io/projected/06f0239a-d786-4bad-a01d-9cb637ffdcd0-kube-api-access-9s2sl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.259272 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.259251    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.312902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.312871    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:45:00.350139 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.350108    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"]
Nov 03 12:45:00.354061 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.354033    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"]
Nov 03 12:45:00.360913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.360885    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvnp\" (UniqueName: \"kubernetes.io/projected/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-kube-api-access-brvnp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.361064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.360934    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86254b03-179d-466c-9ecf-2151c563f8f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.361064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.360998    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdsn\" (UniqueName: \"kubernetes.io/projected/86254b03-179d-466c-9ecf-2151c563f8f9-kube-api-access-4cdsn\") pod \"maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.361064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.361049    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqx5\" (UniqueName: \"kubernetes.io/projected/805422ad-0e4d-4e4b-a625-f622206a878c-kube-api-access-vwqx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.362205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.362183    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/805422ad-0e4d-4e4b-a625-f622206a878c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.363311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.362221    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2sl\" (UniqueName: \"kubernetes.io/projected/06f0239a-d786-4bad-a01d-9cb637ffdcd0-kube-api-access-9s2sl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.363311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.362289    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.363311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.362367    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ee7b15-2d11-41c8-bf55-c25468cadde7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.363311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.362439    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86254b03-179d-466c-9ecf-2151c563f8f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.363311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.362453    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shzd\" (UniqueName: \"kubernetes.io/projected/57ee7b15-2d11-41c8-bf55-c25468cadde7-kube-api-access-8shzd\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.363311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.362483    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f0239a-d786-4bad-a01d-9cb637ffdcd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.363311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.363303    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/805422ad-0e4d-4e4b-a625-f622206a878c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.365011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.363635    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f0239a-d786-4bad-a01d-9cb637ffdcd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.365011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.364399    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ee7b15-2d11-41c8-bf55-c25468cadde7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.365011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.364427    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.370624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.370603    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"]
Nov 03 12:45:00.411315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.411278    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"]
Nov 03 12:45:00.415535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.415505    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"]
Nov 03 12:45:00.417245 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.417219    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdsn\" (UniqueName: \"kubernetes.io/projected/86254b03-179d-466c-9ecf-2151c563f8f9-kube-api-access-4cdsn\") pod \"maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.418551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.418523    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvnp\" (UniqueName: \"kubernetes.io/projected/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-kube-api-access-brvnp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.418692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.418599    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqx5\" (UniqueName: \"kubernetes.io/projected/805422ad-0e4d-4e4b-a625-f622206a878c-kube-api-access-vwqx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.419611 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.419582    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2sl\" (UniqueName: \"kubernetes.io/projected/06f0239a-d786-4bad-a01d-9cb637ffdcd0-kube-api-access-9s2sl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.431085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.431026    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shzd\" (UniqueName: \"kubernetes.io/projected/57ee7b15-2d11-41c8-bf55-c25468cadde7-kube-api-access-8shzd\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.496997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.496971    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:00.503168 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.503147    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:00.515323 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.515301    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:00.540917 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.540880    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:00.550546 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.550517    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:00.876072 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.876049    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"]
Nov 03 12:45:00.877189 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:00.877163    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805422ad_0e4d_4e4b_a625_f622206a878c.slice/crio-09ccb8b566643cad57265244522a9b20ddc4bd70f8685aec4b46ff99bd24af0c WatchSource:0}: Error finding container 09ccb8b566643cad57265244522a9b20ddc4bd70f8685aec4b46ff99bd24af0c: Status 404 returned error can't find the container with id 09ccb8b566643cad57265244522a9b20ddc4bd70f8685aec4b46ff99bd24af0c
Nov 03 12:45:00.923305 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.922854    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"]
Nov 03 12:45:00.923872 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:00.923767    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f0239a_d786_4bad_a01d_9cb637ffdcd0.slice/crio-85dcbd1c74f88e6a8ab61cce5b64e867a10cdd946c3efc1e9487871b5b469875 WatchSource:0}: Error finding container 85dcbd1c74f88e6a8ab61cce5b64e867a10cdd946c3efc1e9487871b5b469875: Status 404 returned error can't find the container with id 85dcbd1c74f88e6a8ab61cce5b64e867a10cdd946c3efc1e9487871b5b469875
Nov 03 12:45:00.926910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.926885    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"]
Nov 03 12:45:00.927103 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:00.927083    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd6aa02_3fd1_4aa9_b6e1_3555526ee59d.slice/crio-01a2826ee777b14c094e7343b90f6c8a66ba4edd6b3e19e46a7af14fe68fdc03 WatchSource:0}: Error finding container 01a2826ee777b14c094e7343b90f6c8a66ba4edd6b3e19e46a7af14fe68fdc03: Status 404 returned error can't find the container with id 01a2826ee777b14c094e7343b90f6c8a66ba4edd6b3e19e46a7af14fe68fdc03
Nov 03 12:45:00.989071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.989044    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"]
Nov 03 12:45:00.990762 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:00.990736    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86254b03_179d_466c_9ecf_2151c563f8f9.slice/crio-badcc1bd966011c203029b591927a623ec5f926fa42da6dde2b417d5cce61541 WatchSource:0}: Error finding container badcc1bd966011c203029b591927a623ec5f926fa42da6dde2b417d5cce61541: Status 404 returned error can't find the container with id badcc1bd966011c203029b591927a623ec5f926fa42da6dde2b417d5cce61541
Nov 03 12:45:00.992206 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:00.992183    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"]
Nov 03 12:45:00.992833 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:00.992814    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ee7b15_2d11_41c8_bf55_c25468cadde7.slice/crio-76db6361a212eb40d691b3c7d018f8b0826234be7410a7d1615a8b51a957bc85 WatchSource:0}: Error finding container 76db6361a212eb40d691b3c7d018f8b0826234be7410a7d1615a8b51a957bc85: Status 404 returned error can't find the container with id 76db6361a212eb40d691b3c7d018f8b0826234be7410a7d1615a8b51a957bc85
Nov 03 12:45:01.017113 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:01.017087    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" event={"ID":"86254b03-179d-466c-9ecf-2151c563f8f9","Type":"ContainerStarted","Data":"badcc1bd966011c203029b591927a623ec5f926fa42da6dde2b417d5cce61541"}
Nov 03 12:45:01.018550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:01.018522    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" event={"ID":"57ee7b15-2d11-41c8-bf55-c25468cadde7","Type":"ContainerStarted","Data":"76db6361a212eb40d691b3c7d018f8b0826234be7410a7d1615a8b51a957bc85"}
Nov 03 12:45:01.019798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:01.019778    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" event={"ID":"06f0239a-d786-4bad-a01d-9cb637ffdcd0","Type":"ContainerStarted","Data":"85dcbd1c74f88e6a8ab61cce5b64e867a10cdd946c3efc1e9487871b5b469875"}
Nov 03 12:45:01.020983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:01.020963    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" event={"ID":"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d","Type":"ContainerStarted","Data":"01a2826ee777b14c094e7343b90f6c8a66ba4edd6b3e19e46a7af14fe68fdc03"}
Nov 03 12:45:01.022171 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:01.022152    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" event={"ID":"805422ad-0e4d-4e4b-a625-f622206a878c","Type":"ContainerStarted","Data":"09ccb8b566643cad57265244522a9b20ddc4bd70f8685aec4b46ff99bd24af0c"}
Nov 03 12:45:01.983524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:01.983497    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"]
Nov 03 12:45:01.986798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:01.986777    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.033252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.033223    4909 generic.go:356] "Generic (PLEG): container finished" podID="805422ad-0e4d-4e4b-a625-f622206a878c" containerID="b436c800c3ee1bc291b27f4d3c478e253fda903f7759413031b93fd2879b9a5c" exitCode=60
Nov 03 12:45:02.033438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.033414    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" event={"ID":"805422ad-0e4d-4e4b-a625-f622206a878c","Type":"ContainerDied","Data":"b436c800c3ee1bc291b27f4d3c478e253fda903f7759413031b93fd2879b9a5c"}
Nov 03 12:45:02.033745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.033689    4909 scope.go:122] "RemoveContainer" containerID="b436c800c3ee1bc291b27f4d3c478e253fda903f7759413031b93fd2879b9a5c"
Nov 03 12:45:02.035092 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.035066    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-6fb3a8-openshift-default-dockercfg-j9znp\""
Nov 03 12:45:02.037997 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.037975    4909 generic.go:356] "Generic (PLEG): container finished" podID="86254b03-179d-466c-9ecf-2151c563f8f9" containerID="e5692a8fb35466ccb6ca71b8b8c12859f74a0038dc2f894d9190f996262788da" exitCode=60
Nov 03 12:45:02.038191 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.038172    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" event={"ID":"86254b03-179d-466c-9ecf-2151c563f8f9","Type":"ContainerDied","Data":"e5692a8fb35466ccb6ca71b8b8c12859f74a0038dc2f894d9190f996262788da"}
Nov 03 12:45:02.038868 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.038849    4909 scope.go:122] "RemoveContainer" containerID="e5692a8fb35466ccb6ca71b8b8c12859f74a0038dc2f894d9190f996262788da"
Nov 03 12:45:02.046544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.046520    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"]
Nov 03 12:45:02.047888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.047867    4909 generic.go:356] "Generic (PLEG): container finished" podID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerID="cd2000d71ec37671291eb665ca5bc46dbf63d8f0b64663def7329a28654e063d" exitCode=60
Nov 03 12:45:02.048004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.047924    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" event={"ID":"57ee7b15-2d11-41c8-bf55-c25468cadde7","Type":"ContainerDied","Data":"cd2000d71ec37671291eb665ca5bc46dbf63d8f0b64663def7329a28654e063d"}
Nov 03 12:45:02.048281 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.048255    4909 scope.go:122] "RemoveContainer" containerID="cd2000d71ec37671291eb665ca5bc46dbf63d8f0b64663def7329a28654e063d"
Nov 03 12:45:02.050342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.050318    4909 generic.go:356] "Generic (PLEG): container finished" podID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerID="ff496c6ad995d90463e7d036e373ba866405dd3cd82069e995e8c796b6fc5570" exitCode=60
Nov 03 12:45:02.050520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.050347    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" event={"ID":"06f0239a-d786-4bad-a01d-9cb637ffdcd0","Type":"ContainerDied","Data":"ff496c6ad995d90463e7d036e373ba866405dd3cd82069e995e8c796b6fc5570"}
Nov 03 12:45:02.050634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.050611    4909 scope.go:122] "RemoveContainer" containerID="ff496c6ad995d90463e7d036e373ba866405dd3cd82069e995e8c796b6fc5570"
Nov 03 12:45:02.055043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.055016    4909 generic.go:356] "Generic (PLEG): container finished" podID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerID="337bdff421a3136f16aef8d4dca0cbdeb9839249dd97c3383024848cd1f7024c" exitCode=60
Nov 03 12:45:02.055250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.055228    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" event={"ID":"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d","Type":"ContainerDied","Data":"337bdff421a3136f16aef8d4dca0cbdeb9839249dd97c3383024848cd1f7024c"}
Nov 03 12:45:02.055740 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.055562    4909 scope.go:122] "RemoveContainer" containerID="337bdff421a3136f16aef8d4dca0cbdeb9839249dd97c3383024848cd1f7024c"
Nov 03 12:45:02.085693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.085665    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086017    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086078    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086357    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679a8bb6-6ac5-455b-a303-0f6201ba1556-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086409    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086600    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086643    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbswv\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-kube-api-access-sbswv\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086722    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.086869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.086785    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.191854    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.191904    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbswv\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-kube-api-access-sbswv\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.191955    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.191998    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.192098    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.192211    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.192235    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.192260    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679a8bb6-6ac5-455b-a303-0f6201ba1556-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.192293    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.196317    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.196740    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.197414    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.197727    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.200237    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679a8bb6-6ac5-455b-a303-0f6201ba1556-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.208976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.205189    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.210181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.210128    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.342446 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.342382    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.342577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.342467    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbswv\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-kube-api-access-sbswv\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.532571 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.532541    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:02.810931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.810897    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2" podUID="3ff7d855-087e-4353-b807-c831d74489fc" containerName="main" containerID="cri-o://036020e869dbfa13c709a42e6f667d135eb3afac737f6507f79b60e1f18b41c1" gracePeriod=45
Nov 03 12:45:02.954218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.954194    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"]
Nov 03 12:45:02.956189 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:02.956158    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679a8bb6_6ac5_455b_a303_0f6201ba1556.slice/crio-ca2aa12bfb4a7c644a764b80b86db4b9ede8772be165446cfe2b6bf6a344808f WatchSource:0}: Error finding container ca2aa12bfb4a7c644a764b80b86db4b9ede8772be165446cfe2b6bf6a344808f: Status 404 returned error can't find the container with id ca2aa12bfb4a7c644a764b80b86db4b9ede8772be165446cfe2b6bf6a344808f
Nov 03 12:45:02.959719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.959670    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:45:02.959812 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.959769    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:45:02.959871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.959813    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:45:02.970180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:02.970144    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2" podUID="3ff7d855-087e-4353-b807-c831d74489fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.155:8000/ready\": dial tcp 10.132.0.155:8000: connect: connection refused"
Nov 03 12:45:03.069107 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.069072    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" event={"ID":"86254b03-179d-466c-9ecf-2151c563f8f9","Type":"ContainerStarted","Data":"ec4f167ce7007762f7a58453514932d49d376a6f8a1ce85df2aa32bd681dc9fd"}
Nov 03 12:45:03.071403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.071377    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" event={"ID":"57ee7b15-2d11-41c8-bf55-c25468cadde7","Type":"ContainerStarted","Data":"4674ad02bae8968c00b0e4f3f5ee465db5e8c74464102620c3e6f478d1e386b4"}
Nov 03 12:45:03.073368 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.073338    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" event={"ID":"06f0239a-d786-4bad-a01d-9cb637ffdcd0","Type":"ContainerStarted","Data":"4be15641e959a1ce122ea9518c1204a78474cbc60122cf0dada6c9e7783c8ab0"}
Nov 03 12:45:03.075461 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.075435    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" event={"ID":"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d","Type":"ContainerStarted","Data":"78250c8ad1a84e40f2183976289211007b304b5e1627d2da4f9985135b93e4c6"}
Nov 03 12:45:03.077136 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.077034    4909 generic.go:356] "Generic (PLEG): container finished" podID="3ff7d855-087e-4353-b807-c831d74489fc" containerID="036020e869dbfa13c709a42e6f667d135eb3afac737f6507f79b60e1f18b41c1" exitCode=0
Nov 03 12:45:03.077136 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.077096    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2" event={"ID":"3ff7d855-087e-4353-b807-c831d74489fc","Type":"ContainerDied","Data":"036020e869dbfa13c709a42e6f667d135eb3afac737f6507f79b60e1f18b41c1"}
Nov 03 12:45:03.078999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.078971    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" event={"ID":"679a8bb6-6ac5-455b-a303-0f6201ba1556","Type":"ContainerStarted","Data":"ca2aa12bfb4a7c644a764b80b86db4b9ede8772be165446cfe2b6bf6a344808f"}
Nov 03 12:45:03.082710 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.081104    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" event={"ID":"805422ad-0e4d-4e4b-a625-f622206a878c","Type":"ContainerStarted","Data":"c797747ee7406b173a5d32cfeb61937021ae1ce359de4ca5e51eab545bcfc164"}
Nov 03 12:45:03.111589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.111566    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:45:03.174369 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.174313    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" podStartSLOduration=3.1742945750000002 podStartE2EDuration="3.174294575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.173526038 +0000 UTC m=+3252.185992715" watchObservedRunningTime="2026-11-03 12:45:03.174294575 +0000 UTC m=+3252.186761242"
Nov 03 12:45:03.205276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205250    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff7d855-087e-4353-b807-c831d74489fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7d855-087e-4353-b807-c831d74489fc-tls-certs\") pod \"3ff7d855-087e-4353-b807-c831d74489fc\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") "
Nov 03 12:45:03.205412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205347    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff7d855-087e-4353-b807-c831d74489fc-kube-api-access-92w4h\" (UniqueName: \"kubernetes.io/projected/3ff7d855-087e-4353-b807-c831d74489fc-kube-api-access-92w4h\") pod \"3ff7d855-087e-4353-b807-c831d74489fc\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") "
Nov 03 12:45:03.205412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205384    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-dshm\") pod \"3ff7d855-087e-4353-b807-c831d74489fc\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") "
Nov 03 12:45:03.205524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205431    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-model-cache\") pod \"3ff7d855-087e-4353-b807-c831d74489fc\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") "
Nov 03 12:45:03.205524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205474    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-tmp-dir\") pod \"3ff7d855-087e-4353-b807-c831d74489fc\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") "
Nov 03 12:45:03.205524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205510    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-home\") pod \"3ff7d855-087e-4353-b807-c831d74489fc\" (UID: \"3ff7d855-087e-4353-b807-c831d74489fc\") "
Nov 03 12:45:03.206336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205750    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-model-cache" pod "3ff7d855-087e-4353-b807-c831d74489fc" (UID: "3ff7d855-087e-4353-b807-c831d74489fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:03.206336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.205948    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-tmp-dir" pod "3ff7d855-087e-4353-b807-c831d74489fc" (UID: "3ff7d855-087e-4353-b807-c831d74489fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:03.206336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.206201    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:03.206336 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.206226    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:03.206726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.206689    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-home" pod "3ff7d855-087e-4353-b807-c831d74489fc" (UID: "3ff7d855-087e-4353-b807-c831d74489fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:03.207784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.207756    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff7d855-087e-4353-b807-c831d74489fc-tls-certs" pod "3ff7d855-087e-4353-b807-c831d74489fc" (UID: "3ff7d855-087e-4353-b807-c831d74489fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:45:03.207866 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.207795    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff7d855-087e-4353-b807-c831d74489fc-kube-api-access-92w4h" pod "3ff7d855-087e-4353-b807-c831d74489fc" (UID: "3ff7d855-087e-4353-b807-c831d74489fc"). InnerVolumeSpecName "kube-api-access-92w4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:03.208052 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.208034    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-dshm" pod "3ff7d855-087e-4353-b807-c831d74489fc" (UID: "3ff7d855-087e-4353-b807-c831d74489fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:03.278676 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.278617    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" podStartSLOduration=3.278604984 podStartE2EDuration="3.278604984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.276991846 +0000 UTC m=+3252.289458508" watchObservedRunningTime="2026-11-03 12:45:03.278604984 +0000 UTC m=+3252.291071647"
Nov 03 12:45:03.307878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.307850    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:03.307878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.307875    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7d855-087e-4353-b807-c831d74489fc-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:03.308026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.307886    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92w4h\" (UniqueName: \"kubernetes.io/projected/3ff7d855-087e-4353-b807-c831d74489fc-kube-api-access-92w4h\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:03.308026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.307896    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7d855-087e-4353-b807-c831d74489fc-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:03.385130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.385074    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" podStartSLOduration=3.385056614 podStartE2EDuration="3.385056614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.380359708 +0000 UTC m=+3252.392826372" watchObservedRunningTime="2026-11-03 12:45:03.385056614 +0000 UTC m=+3252.397523278"
Nov 03 12:45:03.485331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.485286    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" podStartSLOduration=3.485272692 podStartE2EDuration="3.485272692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.483260336 +0000 UTC m=+3252.495726998" watchObservedRunningTime="2026-11-03 12:45:03.485272692 +0000 UTC m=+3252.497739377"
Nov 03 12:45:03.588617 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:03.588567    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" podStartSLOduration=3.588553165 podStartE2EDuration="3.588553165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.587032546 +0000 UTC m=+3252.599499208" watchObservedRunningTime="2026-11-03 12:45:03.588553165 +0000 UTC m=+3252.601019827"
Nov 03 12:45:04.090284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.090249    4909 generic.go:356] "Generic (PLEG): container finished" podID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerID="4674ad02bae8968c00b0e4f3f5ee465db5e8c74464102620c3e6f478d1e386b4" exitCode=60
Nov 03 12:45:04.090811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.090326    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" event={"ID":"57ee7b15-2d11-41c8-bf55-c25468cadde7","Type":"ContainerDied","Data":"4674ad02bae8968c00b0e4f3f5ee465db5e8c74464102620c3e6f478d1e386b4"}
Nov 03 12:45:04.090811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.090387    4909 scope.go:122] "RemoveContainer" containerID="cd2000d71ec37671291eb665ca5bc46dbf63d8f0b64663def7329a28654e063d"
Nov 03 12:45:04.090929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.090864    4909 scope.go:122] "RemoveContainer" containerID="4674ad02bae8968c00b0e4f3f5ee465db5e8c74464102620c3e6f478d1e386b4"
Nov 03 12:45:04.091223 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:04.091192    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g_odh-ai-gateway-infra(57ee7b15-2d11-41c8-bf55-c25468cadde7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7"
Nov 03 12:45:04.092525 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.092500    4909 generic.go:356] "Generic (PLEG): container finished" podID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerID="4be15641e959a1ce122ea9518c1204a78474cbc60122cf0dada6c9e7783c8ab0" exitCode=60
Nov 03 12:45:04.092624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.092549    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" event={"ID":"06f0239a-d786-4bad-a01d-9cb637ffdcd0","Type":"ContainerDied","Data":"4be15641e959a1ce122ea9518c1204a78474cbc60122cf0dada6c9e7783c8ab0"}
Nov 03 12:45:04.092846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.092826    4909 scope.go:122] "RemoveContainer" containerID="4be15641e959a1ce122ea9518c1204a78474cbc60122cf0dada6c9e7783c8ab0"
Nov 03 12:45:04.093106 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:04.093089    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb_odh-ai-gateway-infra(06f0239a-d786-4bad-a01d-9cb637ffdcd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0"
Nov 03 12:45:04.094412 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.094393    4909 generic.go:356] "Generic (PLEG): container finished" podID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerID="78250c8ad1a84e40f2183976289211007b304b5e1627d2da4f9985135b93e4c6" exitCode=60
Nov 03 12:45:04.094519 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.094471    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" event={"ID":"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d","Type":"ContainerDied","Data":"78250c8ad1a84e40f2183976289211007b304b5e1627d2da4f9985135b93e4c6"}
Nov 03 12:45:04.094659 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.094637    4909 scope.go:122] "RemoveContainer" containerID="78250c8ad1a84e40f2183976289211007b304b5e1627d2da4f9985135b93e4c6"
Nov 03 12:45:04.094888 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:04.094869    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74_odh-ai-gateway-infra(8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d"
Nov 03 12:45:04.095970 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.095954    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"
Nov 03 12:45:04.096187 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.095955    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2" event={"ID":"3ff7d855-087e-4353-b807-c831d74489fc","Type":"ContainerDied","Data":"7b3bdb728bd8182caaa3eb9c56e9991ae7667c23d936d4f6b071179e627bd91a"}
Nov 03 12:45:04.097834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.097808    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" event={"ID":"679a8bb6-6ac5-455b-a303-0f6201ba1556","Type":"ContainerStarted","Data":"80c0adcde17e37b6a184fe8633239529d2818b9938c3544e38901513ef078a60"}
Nov 03 12:45:04.099848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.099824    4909 generic.go:356] "Generic (PLEG): container finished" podID="805422ad-0e4d-4e4b-a625-f622206a878c" containerID="c797747ee7406b173a5d32cfeb61937021ae1ce359de4ca5e51eab545bcfc164" exitCode=60
Nov 03 12:45:04.099926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.099906    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" event={"ID":"805422ad-0e4d-4e4b-a625-f622206a878c","Type":"ContainerDied","Data":"c797747ee7406b173a5d32cfeb61937021ae1ce359de4ca5e51eab545bcfc164"}
Nov 03 12:45:04.100135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.100115    4909 scope.go:122] "RemoveContainer" containerID="c797747ee7406b173a5d32cfeb61937021ae1ce359de4ca5e51eab545bcfc164"
Nov 03 12:45:04.100401 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:04.100375    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw_odh-ai-gateway-infra(805422ad-0e4d-4e4b-a625-f622206a878c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" podUID="805422ad-0e4d-4e4b-a625-f622206a878c"
Nov 03 12:45:04.107781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.107762    4909 scope.go:122] "RemoveContainer" containerID="ff496c6ad995d90463e7d036e373ba866405dd3cd82069e995e8c796b6fc5570"
Nov 03 12:45:04.108629 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.108602    4909 generic.go:356] "Generic (PLEG): container finished" podID="86254b03-179d-466c-9ecf-2151c563f8f9" containerID="ec4f167ce7007762f7a58453514932d49d376a6f8a1ce85df2aa32bd681dc9fd" exitCode=60
Nov 03 12:45:04.108752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.108643    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" event={"ID":"86254b03-179d-466c-9ecf-2151c563f8f9","Type":"ContainerDied","Data":"ec4f167ce7007762f7a58453514932d49d376a6f8a1ce85df2aa32bd681dc9fd"}
Nov 03 12:45:04.108930 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.108913    4909 scope.go:122] "RemoveContainer" containerID="ec4f167ce7007762f7a58453514932d49d376a6f8a1ce85df2aa32bd681dc9fd"
Nov 03 12:45:04.109176 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:04.109157    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc_odh-ai-gateway-infra(86254b03-179d-466c-9ecf-2151c563f8f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" podUID="86254b03-179d-466c-9ecf-2151c563f8f9"
Nov 03 12:45:04.119044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.119023    4909 scope.go:122] "RemoveContainer" containerID="337bdff421a3136f16aef8d4dca0cbdeb9839249dd97c3383024848cd1f7024c"
Nov 03 12:45:04.130731 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.130714    4909 scope.go:122] "RemoveContainer" containerID="036020e869dbfa13c709a42e6f667d135eb3afac737f6507f79b60e1f18b41c1"
Nov 03 12:45:04.138948 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.138931    4909 scope.go:122] "RemoveContainer" containerID="b436c800c3ee1bc291b27f4d3c478e253fda903f7759413031b93fd2879b9a5c"
Nov 03 12:45:04.150136 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.150111    4909 scope.go:122] "RemoveContainer" containerID="e5692a8fb35466ccb6ca71b8b8c12859f74a0038dc2f894d9190f996262788da"
Nov 03 12:45:04.510228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.510167    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podStartSLOduration=3.510153883 podStartE2EDuration="3.510153883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:04.508960827 +0000 UTC m=+3253.521427489" watchObservedRunningTime="2026-11-03 12:45:04.510153883 +0000 UTC m=+3253.522620546"
Nov 03 12:45:04.533378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.533350    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:04.539401 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.539381    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:04.718732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.718687    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"]
Nov 03 12:45:04.726813 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.726778    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/auth-test-model-e0615c-kserve-6777569ff7-cl7j2"]
Nov 03 12:45:04.980766 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:04.980735    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:45:05.116709 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.116660    4909 scope.go:122] "RemoveContainer" containerID="ec4f167ce7007762f7a58453514932d49d376a6f8a1ce85df2aa32bd681dc9fd"
Nov 03 12:45:05.117141 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:05.117040    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc_odh-ai-gateway-infra(86254b03-179d-466c-9ecf-2151c563f8f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" podUID="86254b03-179d-466c-9ecf-2151c563f8f9"
Nov 03 12:45:05.118875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.118853    4909 scope.go:122] "RemoveContainer" containerID="4674ad02bae8968c00b0e4f3f5ee465db5e8c74464102620c3e6f478d1e386b4"
Nov 03 12:45:05.119178 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:05.119152    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g_odh-ai-gateway-infra(57ee7b15-2d11-41c8-bf55-c25468cadde7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7"
Nov 03 12:45:05.121234 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.121209    4909 scope.go:122] "RemoveContainer" containerID="4be15641e959a1ce122ea9518c1204a78474cbc60122cf0dada6c9e7783c8ab0"
Nov 03 12:45:05.121495 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:05.121472    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb_odh-ai-gateway-infra(06f0239a-d786-4bad-a01d-9cb637ffdcd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0"
Nov 03 12:45:05.123307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.123281    4909 scope.go:122] "RemoveContainer" containerID="78250c8ad1a84e40f2183976289211007b304b5e1627d2da4f9985135b93e4c6"
Nov 03 12:45:05.123551 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:05.123526    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74_odh-ai-gateway-infra(8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d"
Nov 03 12:45:05.126043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.126023    4909 scope.go:122] "RemoveContainer" containerID="c797747ee7406b173a5d32cfeb61937021ae1ce359de4ca5e51eab545bcfc164"
Nov 03 12:45:05.126286 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:05.126262    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw_odh-ai-gateway-infra(805422ad-0e4d-4e4b-a625-f622206a878c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" podUID="805422ad-0e4d-4e4b-a625-f622206a878c"
Nov 03 12:45:05.126458 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.126439    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff7d855-087e-4353-b807-c831d74489fc" path="/var/lib/kubelet/pods/3ff7d855-087e-4353-b807-c831d74489fc/volumes"
Nov 03 12:45:05.126859 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.126836    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:05.127140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:05.127121    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:45:09.564861 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:09.564826    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:45:11.310291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.310256    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"]
Nov 03 12:45:11.311168 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.311141    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3ff7d855-087e-4353-b807-c831d74489fc" containerName="main"
Nov 03 12:45:11.311168 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.311166    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff7d855-087e-4353-b807-c831d74489fc" containerName="main"
Nov 03 12:45:11.311407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.311391    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff7d855-087e-4353-b807-c831d74489fc" containerName="main"
Nov 03 12:45:11.315989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.315965    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.398755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.391364    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"]
Nov 03 12:45:11.416056 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.416012    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-6fb3a8-dockercfg-98q2b\""
Nov 03 12:45:11.416243 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.416110    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-6fb3a8\""
Nov 03 12:45:11.428174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.424190    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2bm\" (UniqueName: \"kubernetes.io/projected/1e837472-8f8f-4682-ad78-0f7779e164f7-kube-api-access-sm2bm\") pod \"payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.428174 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.424273    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e837472-8f8f-4682-ad78-0f7779e164f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.470929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.470897    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"]
Nov 03 12:45:11.475526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.475502    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:11.525160 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.525128    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e837472-8f8f-4682-ad78-0f7779e164f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.525442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.525417    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2bm\" (UniqueName: \"kubernetes.io/projected/1e837472-8f8f-4682-ad78-0f7779e164f7-kube-api-access-sm2bm\") pod \"payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.526085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.526057    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e837472-8f8f-4682-ad78-0f7779e164f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.528979 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.528952    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"]
Nov 03 12:45:11.578480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.578417    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2bm\" (UniqueName: \"kubernetes.io/projected/1e837472-8f8f-4682-ad78-0f7779e164f7-kube-api-access-sm2bm\") pod \"payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.626350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.626320    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931adf2-8c4f-4539-be64-6d5f17e08d96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:11.626522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.626444    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkss\" (UniqueName: \"kubernetes.io/projected/c931adf2-8c4f-4539-be64-6d5f17e08d96-kube-api-access-5gkss\") pod \"payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:11.629505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.629485    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:11.727847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.727813    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931adf2-8c4f-4539-be64-6d5f17e08d96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:11.728005 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.727974    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkss\" (UniqueName: \"kubernetes.io/projected/c931adf2-8c4f-4539-be64-6d5f17e08d96-kube-api-access-5gkss\") pod \"payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:11.728535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.728516    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931adf2-8c4f-4539-be64-6d5f17e08d96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:11.780609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.780578    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkss\" (UniqueName: \"kubernetes.io/projected/c931adf2-8c4f-4539-be64-6d5f17e08d96-kube-api-access-5gkss\") pod \"payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:11.789362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:11.789343    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:12.000263 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:12.000235    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"]
Nov 03 12:45:12.001155 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:12.001127    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e837472_8f8f_4682_ad78_0f7779e164f7.slice/crio-322d27fa5248cb07c21df63d5c476eb75f8a73c9e3a3ac234241983de49ae7c7 WatchSource:0}: Error finding container 322d27fa5248cb07c21df63d5c476eb75f8a73c9e3a3ac234241983de49ae7c7: Status 404 returned error can't find the container with id 322d27fa5248cb07c21df63d5c476eb75f8a73c9e3a3ac234241983de49ae7c7
Nov 03 12:45:12.155779 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:12.155756    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"]
Nov 03 12:45:12.157670 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:45:12.157641    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc931adf2_8c4f_4539_be64_6d5f17e08d96.slice/crio-8790e627ac36a72e4f369eb8986d252e20321f0b9b211fa75f06ed522ecd30ac WatchSource:0}: Error finding container 8790e627ac36a72e4f369eb8986d252e20321f0b9b211fa75f06ed522ecd30ac: Status 404 returned error can't find the container with id 8790e627ac36a72e4f369eb8986d252e20321f0b9b211fa75f06ed522ecd30ac
Nov 03 12:45:12.169406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:12.169374    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" event={"ID":"1e837472-8f8f-4682-ad78-0f7779e164f7","Type":"ContainerStarted","Data":"cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64"}
Nov 03 12:45:12.169517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:12.169438    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:12.169517 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:12.169465    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" event={"ID":"1e837472-8f8f-4682-ad78-0f7779e164f7","Type":"ContainerStarted","Data":"322d27fa5248cb07c21df63d5c476eb75f8a73c9e3a3ac234241983de49ae7c7"}
Nov 03 12:45:12.170726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:12.170682    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" event={"ID":"c931adf2-8c4f-4539-be64-6d5f17e08d96","Type":"ContainerStarted","Data":"8790e627ac36a72e4f369eb8986d252e20321f0b9b211fa75f06ed522ecd30ac"}
Nov 03 12:45:12.274159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:12.274104    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" podStartSLOduration=1.274086838 podStartE2EDuration="1.274086838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:12.273350685 +0000 UTC m=+3261.285817345" watchObservedRunningTime="2026-11-03 12:45:12.274086838 +0000 UTC m=+3261.286553503"
Nov 03 12:45:13.177599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:13.177549    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" event={"ID":"c931adf2-8c4f-4539-be64-6d5f17e08d96","Type":"ContainerStarted","Data":"90ad301661cdc992b54a633ca207d902b76f06543fad4e2796e1f209482e8b64"}
Nov 03 12:45:13.178002 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:13.177884    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:13.285211 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:13.285161    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" podStartSLOduration=2.2851491250000002 podStartE2EDuration="2.285149125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:13.281848259 +0000 UTC m=+3262.294314926" watchObservedRunningTime="2026-11-03 12:45:13.285149125 +0000 UTC m=+3262.297615789"
Nov 03 12:45:14.116116 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:14.116084    4909 scope.go:122] "RemoveContainer" containerID="4be15641e959a1ce122ea9518c1204a78474cbc60122cf0dada6c9e7783c8ab0"
Nov 03 12:45:15.116237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:15.116208    4909 scope.go:122] "RemoveContainer" containerID="c797747ee7406b173a5d32cfeb61937021ae1ce359de4ca5e51eab545bcfc164"
Nov 03 12:45:15.116712 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:15.116264    4909 scope.go:122] "RemoveContainer" containerID="78250c8ad1a84e40f2183976289211007b304b5e1627d2da4f9985135b93e4c6"
Nov 03 12:45:15.191676 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:15.191652    4909 generic.go:356] "Generic (PLEG): container finished" podID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerID="c682dbe30b90932bfe01bdf87482f6e2422c3f84931c9975613f9555e6e6ddc6" exitCode=60
Nov 03 12:45:15.191809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:15.191734    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" event={"ID":"06f0239a-d786-4bad-a01d-9cb637ffdcd0","Type":"ContainerDied","Data":"c682dbe30b90932bfe01bdf87482f6e2422c3f84931c9975613f9555e6e6ddc6"}
Nov 03 12:45:15.191809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:15.191802    4909 scope.go:122] "RemoveContainer" containerID="4be15641e959a1ce122ea9518c1204a78474cbc60122cf0dada6c9e7783c8ab0"
Nov 03 12:45:15.192145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:15.192130    4909 scope.go:122] "RemoveContainer" containerID="c682dbe30b90932bfe01bdf87482f6e2422c3f84931c9975613f9555e6e6ddc6"
Nov 03 12:45:15.192376 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:15.192356    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb_odh-ai-gateway-infra(06f0239a-d786-4bad-a01d-9cb637ffdcd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0"
Nov 03 12:45:16.116884 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.116858    4909 scope.go:122] "RemoveContainer" containerID="4674ad02bae8968c00b0e4f3f5ee465db5e8c74464102620c3e6f478d1e386b4"
Nov 03 12:45:16.201501 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.201438    4909 generic.go:356] "Generic (PLEG): container finished" podID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerID="aa0cf12bd16f4caf4b787ff0e72ad9bafc4a04490009bcac26d0adb3aa92aede" exitCode=60
Nov 03 12:45:16.201616 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.201513    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" event={"ID":"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d","Type":"ContainerDied","Data":"aa0cf12bd16f4caf4b787ff0e72ad9bafc4a04490009bcac26d0adb3aa92aede"}
Nov 03 12:45:16.201616 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.201569    4909 scope.go:122] "RemoveContainer" containerID="78250c8ad1a84e40f2183976289211007b304b5e1627d2da4f9985135b93e4c6"
Nov 03 12:45:16.201944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.201923    4909 scope.go:122] "RemoveContainer" containerID="aa0cf12bd16f4caf4b787ff0e72ad9bafc4a04490009bcac26d0adb3aa92aede"
Nov 03 12:45:16.202155 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:16.202136    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74_odh-ai-gateway-infra(8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d"
Nov 03 12:45:16.203561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.203543    4909 generic.go:356] "Generic (PLEG): container finished" podID="805422ad-0e4d-4e4b-a625-f622206a878c" containerID="b583bb095a159375ed7bedf0f714ee0d5c446f262a17ec7d9b890ede5cb8c07d" exitCode=60
Nov 03 12:45:16.203654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.203618    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" event={"ID":"805422ad-0e4d-4e4b-a625-f622206a878c","Type":"ContainerDied","Data":"b583bb095a159375ed7bedf0f714ee0d5c446f262a17ec7d9b890ede5cb8c07d"}
Nov 03 12:45:16.203924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.203910    4909 scope.go:122] "RemoveContainer" containerID="b583bb095a159375ed7bedf0f714ee0d5c446f262a17ec7d9b890ede5cb8c07d"
Nov 03 12:45:16.204104 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:16.204088    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw_odh-ai-gateway-infra(805422ad-0e4d-4e4b-a625-f622206a878c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" podUID="805422ad-0e4d-4e4b-a625-f622206a878c"
Nov 03 12:45:16.214565 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.214551    4909 scope.go:122] "RemoveContainer" containerID="c797747ee7406b173a5d32cfeb61937021ae1ce359de4ca5e51eab545bcfc164"
Nov 03 12:45:16.305158 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.305130    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"]
Nov 03 12:45:16.468574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.468519    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:16.487017 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.486993    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f0239a-d786-4bad-a01d-9cb637ffdcd0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f0239a-d786-4bad-a01d-9cb637ffdcd0-maas-api-service-ca\") pod \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") "
Nov 03 12:45:16.487129 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.487065    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f0239a-d786-4bad-a01d-9cb637ffdcd0-kube-api-access-9s2sl\" (UniqueName: \"kubernetes.io/projected/06f0239a-d786-4bad-a01d-9cb637ffdcd0-kube-api-access-9s2sl\") pod \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\" (UID: \"06f0239a-d786-4bad-a01d-9cb637ffdcd0\") "
Nov 03 12:45:16.487382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.487353    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f0239a-d786-4bad-a01d-9cb637ffdcd0-maas-api-service-ca" pod "06f0239a-d786-4bad-a01d-9cb637ffdcd0" (UID: "06f0239a-d786-4bad-a01d-9cb637ffdcd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:16.489039 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.489017    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f0239a-d786-4bad-a01d-9cb637ffdcd0-kube-api-access-9s2sl" pod "06f0239a-d786-4bad-a01d-9cb637ffdcd0" (UID: "06f0239a-d786-4bad-a01d-9cb637ffdcd0"). InnerVolumeSpecName "kube-api-access-9s2sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:16.588920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.588892    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f0239a-d786-4bad-a01d-9cb637ffdcd0-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:16.588920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:16.588917    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2sl\" (UniqueName: \"kubernetes.io/projected/06f0239a-d786-4bad-a01d-9cb637ffdcd0-kube-api-access-9s2sl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:17.211249 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.211176    4909 generic.go:356] "Generic (PLEG): container finished" podID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerID="d2c0dc279e7a1f180941de180d0f78faaf4c441abb2486b96ee8c7f95d05524f" exitCode=60
Nov 03 12:45:17.211665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.211248    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" event={"ID":"57ee7b15-2d11-41c8-bf55-c25468cadde7","Type":"ContainerDied","Data":"d2c0dc279e7a1f180941de180d0f78faaf4c441abb2486b96ee8c7f95d05524f"}
Nov 03 12:45:17.211665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.211318    4909 scope.go:122] "RemoveContainer" containerID="4674ad02bae8968c00b0e4f3f5ee465db5e8c74464102620c3e6f478d1e386b4"
Nov 03 12:45:17.211833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.211667    4909 scope.go:122] "RemoveContainer" containerID="d2c0dc279e7a1f180941de180d0f78faaf4c441abb2486b96ee8c7f95d05524f"
Nov 03 12:45:17.211946 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:17.211920    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g_odh-ai-gateway-infra(57ee7b15-2d11-41c8-bf55-c25468cadde7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7"
Nov 03 12:45:17.212678 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.212660    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"
Nov 03 12:45:17.212754 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.212664    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb" event={"ID":"06f0239a-d786-4bad-a01d-9cb637ffdcd0","Type":"ContainerDied","Data":"85dcbd1c74f88e6a8ab61cce5b64e867a10cdd946c3efc1e9487871b5b469875"}
Nov 03 12:45:17.224674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.224655    4909 scope.go:122] "RemoveContainer" containerID="c682dbe30b90932bfe01bdf87482f6e2422c3f84931c9975613f9555e6e6ddc6"
Nov 03 12:45:17.321801 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.321779    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"]
Nov 03 12:45:17.429744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.423091    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"]
Nov 03 12:45:17.429744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.427764    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895165-6m2bb"]
Nov 03 12:45:17.471638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.471583    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:17.502497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.502458    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-kube-api-access-brvnp\" (UniqueName: \"kubernetes.io/projected/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-kube-api-access-brvnp\") pod \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") "
Nov 03 12:45:17.502630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.502588    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-maas-api-service-ca\") pod \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\" (UID: \"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d\") "
Nov 03 12:45:17.502933 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.502911    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-maas-api-service-ca" pod "8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" (UID: "8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:17.503043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.503024    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:17.504474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.504451    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-kube-api-access-brvnp" pod "8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" (UID: "8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d"). InnerVolumeSpecName "kube-api-access-brvnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:17.526356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.526333    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"]
Nov 03 12:45:17.604603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.604572    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvnp\" (UniqueName: \"kubernetes.io/projected/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d-kube-api-access-brvnp\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:17.684899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.684873    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:17.704995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.704974    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805422ad-0e4d-4e4b-a625-f622206a878c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/805422ad-0e4d-4e4b-a625-f622206a878c-maas-api-service-ca\") pod \"805422ad-0e4d-4e4b-a625-f622206a878c\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") "
Nov 03 12:45:17.705241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.705220    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805422ad-0e4d-4e4b-a625-f622206a878c-kube-api-access-vwqx5\" (UniqueName: \"kubernetes.io/projected/805422ad-0e4d-4e4b-a625-f622206a878c-kube-api-access-vwqx5\") pod \"805422ad-0e4d-4e4b-a625-f622206a878c\" (UID: \"805422ad-0e4d-4e4b-a625-f622206a878c\") "
Nov 03 12:45:17.705340 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.705319    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805422ad-0e4d-4e4b-a625-f622206a878c-maas-api-service-ca" pod "805422ad-0e4d-4e4b-a625-f622206a878c" (UID: "805422ad-0e4d-4e4b-a625-f622206a878c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:17.705569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.705554    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/805422ad-0e4d-4e4b-a625-f622206a878c-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:17.707223 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.707202    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805422ad-0e4d-4e4b-a625-f622206a878c-kube-api-access-vwqx5" pod "805422ad-0e4d-4e4b-a625-f622206a878c" (UID: "805422ad-0e4d-4e4b-a625-f622206a878c"). InnerVolumeSpecName "kube-api-access-vwqx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:17.807128 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:17.807060    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwqx5\" (UniqueName: \"kubernetes.io/projected/805422ad-0e4d-4e4b-a625-f622206a878c-kube-api-access-vwqx5\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.116102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.116030    4909 scope.go:122] "RemoveContainer" containerID="ec4f167ce7007762f7a58453514932d49d376a6f8a1ce85df2aa32bd681dc9fd"
Nov 03 12:45:18.223486 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.223457    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"
Nov 03 12:45:18.223878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.223503    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw" event={"ID":"805422ad-0e4d-4e4b-a625-f622206a878c","Type":"ContainerDied","Data":"09ccb8b566643cad57265244522a9b20ddc4bd70f8685aec4b46ff99bd24af0c"}
Nov 03 12:45:18.223878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.223570    4909 scope.go:122] "RemoveContainer" containerID="b583bb095a159375ed7bedf0f714ee0d5c446f262a17ec7d9b890ede5cb8c07d"
Nov 03 12:45:18.227773 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.227752    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74" event={"ID":"8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d","Type":"ContainerDied","Data":"01a2826ee777b14c094e7343b90f6c8a66ba4edd6b3e19e46a7af14fe68fdc03"}
Nov 03 12:45:18.227895 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.227801    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"
Nov 03 12:45:18.233922 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.233908    4909 scope.go:122] "RemoveContainer" containerID="aa0cf12bd16f4caf4b787ff0e72ad9bafc4a04490009bcac26d0adb3aa92aede"
Nov 03 12:45:18.325761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.325732    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"]
Nov 03 12:45:18.379351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.379263    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"]
Nov 03 12:45:18.383349 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.382622    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895165-slw74"]
Nov 03 12:45:18.482403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.482374    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"]
Nov 03 12:45:18.487350 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.487321    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895165-bdscw"]
Nov 03 12:45:18.567063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.567043    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:18.616454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.616431    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ee7b15-2d11-41c8-bf55-c25468cadde7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ee7b15-2d11-41c8-bf55-c25468cadde7-maas-api-service-ca\") pod \"57ee7b15-2d11-41c8-bf55-c25468cadde7\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") "
Nov 03 12:45:18.616576 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.616472    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ee7b15-2d11-41c8-bf55-c25468cadde7-kube-api-access-8shzd\" (UniqueName: \"kubernetes.io/projected/57ee7b15-2d11-41c8-bf55-c25468cadde7-kube-api-access-8shzd\") pod \"57ee7b15-2d11-41c8-bf55-c25468cadde7\" (UID: \"57ee7b15-2d11-41c8-bf55-c25468cadde7\") "
Nov 03 12:45:18.616844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.616821    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ee7b15-2d11-41c8-bf55-c25468cadde7-maas-api-service-ca" pod "57ee7b15-2d11-41c8-bf55-c25468cadde7" (UID: "57ee7b15-2d11-41c8-bf55-c25468cadde7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:18.617041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.617027    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ee7b15-2d11-41c8-bf55-c25468cadde7-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.618557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.618538    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ee7b15-2d11-41c8-bf55-c25468cadde7-kube-api-access-8shzd" pod "57ee7b15-2d11-41c8-bf55-c25468cadde7" (UID: "57ee7b15-2d11-41c8-bf55-c25468cadde7"). InnerVolumeSpecName "kube-api-access-8shzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:18.718083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:18.718055    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shzd\" (UniqueName: \"kubernetes.io/projected/57ee7b15-2d11-41c8-bf55-c25468cadde7-kube-api-access-8shzd\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:19.124881 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.124848    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" path="/var/lib/kubelet/pods/06f0239a-d786-4bad-a01d-9cb637ffdcd0/volumes"
Nov 03 12:45:19.125312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.125296    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" path="/var/lib/kubelet/pods/805422ad-0e4d-4e4b-a625-f622206a878c/volumes"
Nov 03 12:45:19.125660 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.125648    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" path="/var/lib/kubelet/pods/8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d/volumes"
Nov 03 12:45:19.237831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.237803    4909 generic.go:356] "Generic (PLEG): container finished" podID="86254b03-179d-466c-9ecf-2151c563f8f9" containerID="fa5ec30994a971b3015760d00c1e7ad984db851a70e348a7bdf4d2592851e3bc" exitCode=60
Nov 03 12:45:19.238254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.237872    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" event={"ID":"86254b03-179d-466c-9ecf-2151c563f8f9","Type":"ContainerDied","Data":"fa5ec30994a971b3015760d00c1e7ad984db851a70e348a7bdf4d2592851e3bc"}
Nov 03 12:45:19.238254 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.237945    4909 scope.go:122] "RemoveContainer" containerID="ec4f167ce7007762f7a58453514932d49d376a6f8a1ce85df2aa32bd681dc9fd"
Nov 03 12:45:19.238362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.238315    4909 scope.go:122] "RemoveContainer" containerID="fa5ec30994a971b3015760d00c1e7ad984db851a70e348a7bdf4d2592851e3bc"
Nov 03 12:45:19.238611 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:19.238585    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc_odh-ai-gateway-infra(86254b03-179d-466c-9ecf-2151c563f8f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" podUID="86254b03-179d-466c-9ecf-2151c563f8f9"
Nov 03 12:45:19.239181 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.239162    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g" event={"ID":"57ee7b15-2d11-41c8-bf55-c25468cadde7","Type":"ContainerDied","Data":"76db6361a212eb40d691b3c7d018f8b0826234be7410a7d1615a8b51a957bc85"}
Nov 03 12:45:19.239255 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.239165    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"
Nov 03 12:45:19.253642 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.253625    4909 scope.go:122] "RemoveContainer" containerID="d2c0dc279e7a1f180941de180d0f78faaf4c441abb2486b96ee8c7f95d05524f"
Nov 03 12:45:19.447125 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.447098    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"]
Nov 03 12:45:19.456857 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:19.455370    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895165-54w5g"]
Nov 03 12:45:20.354182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.354142    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"]
Nov 03 12:45:20.449912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.449882    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:45:20.522871 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.522845    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:20.537524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.537499    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86254b03-179d-466c-9ecf-2151c563f8f9-kube-api-access-4cdsn\" (UniqueName: \"kubernetes.io/projected/86254b03-179d-466c-9ecf-2151c563f8f9-kube-api-access-4cdsn\") pod \"86254b03-179d-466c-9ecf-2151c563f8f9\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") "
Nov 03 12:45:20.537638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.537613    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86254b03-179d-466c-9ecf-2151c563f8f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86254b03-179d-466c-9ecf-2151c563f8f9-maas-api-service-ca\") pod \"86254b03-179d-466c-9ecf-2151c563f8f9\" (UID: \"86254b03-179d-466c-9ecf-2151c563f8f9\") "
Nov 03 12:45:20.538015 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.537990    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86254b03-179d-466c-9ecf-2151c563f8f9-maas-api-service-ca" pod "86254b03-179d-466c-9ecf-2151c563f8f9" (UID: "86254b03-179d-466c-9ecf-2151c563f8f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:20.538285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.538261    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86254b03-179d-466c-9ecf-2151c563f8f9-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:20.539585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.539565    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86254b03-179d-466c-9ecf-2151c563f8f9-kube-api-access-4cdsn" pod "86254b03-179d-466c-9ecf-2151c563f8f9" (UID: "86254b03-179d-466c-9ecf-2151c563f8f9"). InnerVolumeSpecName "kube-api-access-4cdsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:20.640041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:20.639976    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdsn\" (UniqueName: \"kubernetes.io/projected/86254b03-179d-466c-9ecf-2151c563f8f9-kube-api-access-4cdsn\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:21.124242 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:21.124207    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" path="/var/lib/kubelet/pods/57ee7b15-2d11-41c8-bf55-c25468cadde7/volumes"
Nov 03 12:45:21.261589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:21.261564    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"
Nov 03 12:45:21.261756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:21.261600    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc" event={"ID":"86254b03-179d-466c-9ecf-2151c563f8f9","Type":"ContainerDied","Data":"badcc1bd966011c203029b591927a623ec5f926fa42da6dde2b417d5cce61541"}
Nov 03 12:45:21.261756 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:21.261659    4909 scope.go:122] "RemoveContainer" containerID="fa5ec30994a971b3015760d00c1e7ad984db851a70e348a7bdf4d2592851e3bc"
Nov 03 12:45:21.416826 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:21.416734    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"]
Nov 03 12:45:21.420483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:21.420446    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52ab15-29895165-lsbxc"]
Nov 03 12:45:23.124132 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:23.124098    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" path="/var/lib/kubelet/pods/86254b03-179d-466c-9ecf-2151c563f8f9/volumes"
Nov 03 12:45:23.181308 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:23.181280    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 12:45:24.186849 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:24.186819    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 12:45:27.992851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:27.992819    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"]
Nov 03 12:45:28.093563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:28.093523    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"]
Nov 03 12:45:32.994179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:32.994064    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr" podUID="86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" containerName="payload-processing" containerID="cri-o://92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f" gracePeriod=25
Nov 03 12:45:33.094569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.094518    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g" podUID="72efce80-b100-4822-ad58-4cb7f99e626e" containerName="payload-pre-processing" containerID="cri-o://c169af9fbfe6dfc2e8c85148c5813b24dc574bb6e0773e5be973d426a1ae5c7c" gracePeriod=25
Nov 03 12:45:33.266732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.266691    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:45:33.361588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.361561    4909 generic.go:356] "Generic (PLEG): container finished" podID="86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" containerID="92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f" exitCode=0
Nov 03 12:45:33.361733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.361599    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"
Nov 03 12:45:33.361733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.361642    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr" event={"ID":"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9","Type":"ContainerDied","Data":"92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f"}
Nov 03 12:45:33.361733 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.361723    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr" event={"ID":"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9","Type":"ContainerDied","Data":"91629b1a3384b726f016a2dc9555b31c8808319a25aa432257d991f34e92ffe7"}
Nov 03 12:45:33.361971 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.361760    4909 scope.go:122] "RemoveContainer" containerID="92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f"
Nov 03 12:45:33.363050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.363031    4909 generic.go:356] "Generic (PLEG): container finished" podID="72efce80-b100-4822-ad58-4cb7f99e626e" containerID="c169af9fbfe6dfc2e8c85148c5813b24dc574bb6e0773e5be973d426a1ae5c7c" exitCode=0
Nov 03 12:45:33.363125 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.363083    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g" event={"ID":"72efce80-b100-4822-ad58-4cb7f99e626e","Type":"ContainerDied","Data":"c169af9fbfe6dfc2e8c85148c5813b24dc574bb6e0773e5be973d426a1ae5c7c"}
Nov 03 12:45:33.377048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.377025    4909 scope.go:122] "RemoveContainer" containerID="92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f"
Nov 03 12:45:33.377819 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:33.377797    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f\": container with ID starting with 92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f not found: ID does not exist" containerID="92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f"
Nov 03 12:45:33.377982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.377958    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f"} err="failed to get container status \"92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f\": rpc error: code = NotFound desc = could not find container \"92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f\": container with ID starting with 92e466c01eb29987c236537f60e9945ad96f53e9e83eb12760ab700e14bbd79f not found: ID does not exist"
Nov 03 12:45:33.377982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.377916    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:45:33.429693 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.425859    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-kube-api-access-4lcrq\" (UniqueName: \"kubernetes.io/projected/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-kube-api-access-4lcrq\") pod \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") "
Nov 03 12:45:33.430408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.430385    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-plugins-config-volume\") pod \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\" (UID: \"86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9\") "
Nov 03 12:45:33.431600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.431532    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-plugins-config-volume" pod "86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" (UID: "86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:33.439790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.435294    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-kube-api-access-4lcrq" pod "86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" (UID: "86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9"). InnerVolumeSpecName "kube-api-access-4lcrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:33.531858 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.531763    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72efce80-b100-4822-ad58-4cb7f99e626e-kube-api-access-sxtn7\" (UniqueName: \"kubernetes.io/projected/72efce80-b100-4822-ad58-4cb7f99e626e-kube-api-access-sxtn7\") pod \"72efce80-b100-4822-ad58-4cb7f99e626e\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") "
Nov 03 12:45:33.532051 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.532031    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72efce80-b100-4822-ad58-4cb7f99e626e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72efce80-b100-4822-ad58-4cb7f99e626e-plugins-config-volume\") pod \"72efce80-b100-4822-ad58-4cb7f99e626e\" (UID: \"72efce80-b100-4822-ad58-4cb7f99e626e\") "
Nov 03 12:45:33.532476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.532445    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72efce80-b100-4822-ad58-4cb7f99e626e-plugins-config-volume" pod "72efce80-b100-4822-ad58-4cb7f99e626e" (UID: "72efce80-b100-4822-ad58-4cb7f99e626e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:33.532783 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.532763    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:33.532889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.532788    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72efce80-b100-4822-ad58-4cb7f99e626e-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:33.532889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.532805    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lcrq\" (UniqueName: \"kubernetes.io/projected/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9-kube-api-access-4lcrq\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:33.534252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.534221    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72efce80-b100-4822-ad58-4cb7f99e626e-kube-api-access-sxtn7" pod "72efce80-b100-4822-ad58-4cb7f99e626e" (UID: "72efce80-b100-4822-ad58-4cb7f99e626e"). InnerVolumeSpecName "kube-api-access-sxtn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:33.634285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.634254    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtn7\" (UniqueName: \"kubernetes.io/projected/72efce80-b100-4822-ad58-4cb7f99e626e-kube-api-access-sxtn7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:33.773372 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.773340    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"]
Nov 03 12:45:33.777134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:33.777110    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52ab15-67cdb8874f-z99fr"]
Nov 03 12:45:34.372535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:34.372461    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"
Nov 03 12:45:34.372535 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:34.372498    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g" event={"ID":"72efce80-b100-4822-ad58-4cb7f99e626e","Type":"ContainerDied","Data":"abf36d0fbfa85bf530fdf007972987ad6b1bc4f7069a19fae7b3ec99698d1ccd"}
Nov 03 12:45:34.373023 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:34.372552    4909 scope.go:122] "RemoveContainer" containerID="c169af9fbfe6dfc2e8c85148c5813b24dc574bb6e0773e5be973d426a1ae5c7c"
Nov 03 12:45:34.523819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:34.523789    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"]
Nov 03 12:45:34.567963 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:34.567933    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52ab15-7d5c85967-qzk8g"]
Nov 03 12:45:35.123451 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:35.123421    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72efce80-b100-4822-ad58-4cb7f99e626e" path="/var/lib/kubelet/pods/72efce80-b100-4822-ad58-4cb7f99e626e/volumes"
Nov 03 12:45:35.123837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:35.123822    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" path="/var/lib/kubelet/pods/86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9/volumes"
Nov 03 12:45:39.564134 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:45:39.564095    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:45:51.545680 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:51.545650    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:45:51.552627 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:51.552599    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:45:54.666311 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:54.666276    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:45:58.682103 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682066    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"]
Nov 03 12:45:58.682906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682889    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682908    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682924    4909 cpu_manager.go:414] "containerMap: removing container" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682931    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682947    4909 cpu_manager.go:414] "containerMap: removing container" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682954    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682976    4909 cpu_manager.go:414] "containerMap: removing container" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682983    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.682996    4909 cpu_manager.go:414] "containerMap: removing container" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:45:58.683007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683002    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683022    4909 cpu_manager.go:414] "containerMap: removing container" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683029    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683040    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683047    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683059    4909 cpu_manager.go:414] "containerMap: removing container" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683066    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683077    4909 cpu_manager.go:414] "containerMap: removing container" podUID="86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" containerName="payload-processing"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683085    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" containerName="payload-processing"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683101    4909 cpu_manager.go:414] "containerMap: removing container" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683107    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683123    4909 cpu_manager.go:414] "containerMap: removing container" podUID="72efce80-b100-4822-ad58-4cb7f99e626e" containerName="payload-pre-processing"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683130    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72efce80-b100-4822-ad58-4cb7f99e626e" containerName="payload-pre-processing"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683141    4909 cpu_manager.go:414] "containerMap: removing container" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683147    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683163    4909 cpu_manager.go:414] "containerMap: removing container" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683170    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683303    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683314    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683328    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683337    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683352    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683361    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683368    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c80eaf-c0dc-4bbf-87c5-67b1faa5ffc9" containerName="payload-processing"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683376    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683383    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683391    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683399    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="72efce80-b100-4822-ad58-4cb7f99e626e" containerName="payload-pre-processing"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683409    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683421    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683431    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:45:58.683463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.683439    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="86254b03-179d-466c-9ecf-2151c563f8f9" containerName="cleanup"
Nov 03 12:45:58.688897 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.688878    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.725886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.725846    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-credential-socket\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.726377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.726170    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-podinfo\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.726671 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.726651    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-token\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.726831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.726813    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-data\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.727481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.727336    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-socket\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.727839 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.727794    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-certs\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.727950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.727881    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-envoy\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.728100 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.728048    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istiod-ca-cert\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.728202 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.728099    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngcxl\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-kube-api-access-ngcxl\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.768276 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.768224    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-94e247-openshift-default-dockercfg-qmggx\""
Nov 03 12:45:58.818496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.818462    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"]
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.835881    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-credential-socket\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.835977    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-podinfo\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.836023    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-token\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.836047    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-data\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.836075    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-socket\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.836107    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-certs\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.836167    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-envoy\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.836216    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istiod-ca-cert\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.836241    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngcxl\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-kube-api-access-ngcxl\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.844725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.837062    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-credential-socket\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.870943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.848495    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-socket\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.871226 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.853348    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-data\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.872306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.872277    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istiod-ca-cert\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.872822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.872800    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-certs\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.928946 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.928911    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngcxl\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-kube-api-access-ngcxl\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.931141 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.931111    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-podinfo\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.955098 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.955001    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-envoy\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:58.981264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:58.981238    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-token\") pod \"e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") " pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:59.008645 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.008617    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:45:59.469134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.465236    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"]
Nov 03 12:45:59.471896 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.471861    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:45:59.472018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.471956    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:45:59.472018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.472001    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:45:59.561723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.560910    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" event={"ID":"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6","Type":"ContainerStarted","Data":"963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94"}
Nov 03 12:45:59.561723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.560982    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" event={"ID":"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6","Type":"ContainerStarted","Data":"6938525873ec4a179294ae1d9a597f5fabf820876b8cdb83347d32e8cfa55ed8"}
Nov 03 12:45:59.669511 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:45:59.669447    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podStartSLOduration=1.669427687 podStartE2EDuration="1.669427687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:59.669404222 +0000 UTC m=+3308.681870884" watchObservedRunningTime="2026-11-03 12:45:59.669427687 +0000 UTC m=+3308.681894352"
Nov 03 12:46:00.009746 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:00.009685    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:46:00.011448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:00.011416    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 12:46:00.011571 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:00.011466    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 12:46:01.013850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:01.013815    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:46:01.576474 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:01.576440    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:46:01.577429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:01.577412    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:46:05.610513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:05.610475    4909 generic.go:356] "Generic (PLEG): container finished" podID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerID="705fe81b0fb806238ac92568283ac90ba5006f9a202d1a61161383daa5b61f89" exitCode=0
Nov 03 12:46:05.611007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:05.610549    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" event={"ID":"a01fbd75-25d5-4efd-8788-44a86c84cf1f","Type":"ContainerDied","Data":"705fe81b0fb806238ac92568283ac90ba5006f9a202d1a61161383daa5b61f89"}
Nov 03 12:46:05.611007 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:05.610984    4909 scope.go:122] "RemoveContainer" containerID="705fe81b0fb806238ac92568283ac90ba5006f9a202d1a61161383daa5b61f89"
Nov 03 12:46:05.612618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:05.612588    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:46:05.612759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:05.612657    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:46:05.612759 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:05.612691    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:46:06.620096 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:06.620052    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" event={"ID":"a01fbd75-25d5-4efd-8788-44a86c84cf1f","Type":"ContainerStarted","Data":"8a9c41c590ca9acb7aeb3cc7b2f05763b2ff228bb69643a9522199bae9a97780"}
Nov 03 12:46:06.675244 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:06.675209    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:46:06.675457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:06.675408    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 12:46:06.675522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:06.675456    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 12:46:07.382941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:07.382905    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"]
Nov 03 12:46:07.383442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:07.383408    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy" containerID="cri-o://871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d" gracePeriod=30
Nov 03 12:46:07.675655 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:07.675562    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 12:46:07.675655 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:07.675611    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 12:46:08.713757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:08.709025    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 12:46:08.713757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:08.709081    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 12:46:09.585130 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:46:09.585092    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:46:09.675407 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:09.675374    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 12:46:09.675596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:09.675425    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 12:46:10.675496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:10.675458    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 12:46:10.675925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:10.675524    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 12:46:11.675328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:11.675295    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 12:46:11.675509 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:11.675356    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 12:46:11.855240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:11.855203    4909 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:46:11.855661 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:11.855264    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:46:12.658744 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.658675    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:46:12.680664 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.680637    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:46:12.681061 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.681043    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:46:12.681914 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.681893    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:46:12.700926 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.700904    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-data\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.700944    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-istio-token\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-istio-token\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701060 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701003    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04708f3d-162a-4077-af9d-ab6d62b12040-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04708f3d-162a-4077-af9d-ab6d62b12040-istio-podinfo\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701156    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-credential-socket\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701208    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-data" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.701250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701240    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-envoy\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701345 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701333    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-kube-api-access-fjrdt\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-kube-api-access-fjrdt\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701386    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-certs\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701412    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-socket\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701441    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-credential-socket" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.701563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701445    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04708f3d-162a-4077-af9d-ab6d62b12040-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04708f3d-162a-4077-af9d-ab6d62b12040-istiod-ca-cert\") pod \"04708f3d-162a-4077-af9d-ab6d62b12040\" (UID: \"04708f3d-162a-4077-af9d-ab6d62b12040\") "
Nov 03 12:46:12.701796 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701777    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04708f3d-162a-4077-af9d-ab6d62b12040-istiod-ca-cert" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:46:12.701937 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701802    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-socket" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.702123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.701878    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-certs" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.702274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.702202    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.702274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.702223    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.702274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.702241    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04708f3d-162a-4077-af9d-ab6d62b12040-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.704070 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.704042    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04708f3d-162a-4077-af9d-ab6d62b12040-istio-podinfo" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:46:12.704152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.704061    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-envoy" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.704152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.704073    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-kube-api-access-fjrdt" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "kube-api-access-fjrdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:46:12.704286 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.704267    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-istio-token" pod "04708f3d-162a-4077-af9d-ab6d62b12040" (UID: "04708f3d-162a-4077-af9d-ab6d62b12040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:46:12.780651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.780620    4909 generic.go:356] "Generic (PLEG): container finished" podID="04708f3d-162a-4077-af9d-ab6d62b12040" containerID="871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d" exitCode=0
Nov 03 12:46:12.780827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.780666    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"
Nov 03 12:46:12.780827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.780712    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" event={"ID":"04708f3d-162a-4077-af9d-ab6d62b12040","Type":"ContainerDied","Data":"871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d"}
Nov 03 12:46:12.780827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.780774    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc" event={"ID":"04708f3d-162a-4077-af9d-ab6d62b12040","Type":"ContainerDied","Data":"c71216be0f800d78591910c3c2e986c3549a69bdb7dfb65a159bf6c5e5c22317"}
Nov 03 12:46:12.780827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.780803    4909 scope.go:122] "RemoveContainer" containerID="871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d"
Nov 03 12:46:12.792347 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.792329    4909 scope.go:122] "RemoveContainer" containerID="efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834"
Nov 03 12:46:12.802533 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.802512    4909 scope.go:122] "RemoveContainer" containerID="871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d"
Nov 03 12:46:12.802832 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:46:12.802809    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d\": container with ID starting with 871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d not found: ID does not exist" containerID="871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d"
Nov 03 12:46:12.802950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.802845    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d"} err="failed to get container status \"871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d\": rpc error: code = NotFound desc = could not find container \"871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d\": container with ID starting with 871d586e6ff6ac50b81476e5d90c613fa94ff3c5c7ebe1eaea2766c81d39a74d not found: ID does not exist"
Nov 03 12:46:12.802950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.802864    4909 scope.go:122] "RemoveContainer" containerID="efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834"
Nov 03 12:46:12.803173 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:46:12.803154    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834\": container with ID starting with efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834 not found: ID does not exist" containerID="efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834"
Nov 03 12:46:12.803264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.803179    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834"} err="failed to get container status \"efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834\": rpc error: code = NotFound desc = could not find container \"efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834\": container with ID starting with efdd315f7526b1f4d025ff4db6b9fa8443c50c93b758b18cdb95ace353995834 not found: ID does not exist"
Nov 03 12:46:12.803516 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.803484    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.803628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.803516    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.803628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.803533    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.803628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.803548    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04708f3d-162a-4077-af9d-ab6d62b12040-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.803628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.803562    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04708f3d-162a-4077-af9d-ab6d62b12040-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.803628 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.803577    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjrdt\" (UniqueName: \"kubernetes.io/projected/04708f3d-162a-4077-af9d-ab6d62b12040-kube-api-access-fjrdt\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.977237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.977184    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"]
Nov 03 12:46:12.982160 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:12.982135    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-8cxfc"]
Nov 03 12:46:13.122965 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:13.122932    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" path="/var/lib/kubelet/pods/04708f3d-162a-4077-af9d-ab6d62b12040/volumes"
Nov 03 12:46:25.767050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:46:25.767016    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:46:39.564290 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:46:39.564256    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:47:05.244003 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.243965    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"]
Nov 03 12:47:05.244641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244618    4909 cpu_manager.go:414] "containerMap: removing container" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244638    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805422ad-0e4d-4e4b-a625-f622206a878c" containerName="cleanup"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244684    4909 cpu_manager.go:414] "containerMap: removing container" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244692    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244725    4909 cpu_manager.go:414] "containerMap: removing container" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244731    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244750    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244756    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244779    4909 cpu_manager.go:414] "containerMap: removing container" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:47:05.244827 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244784    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ee7b15-2d11-41c8-bf55-c25468cadde7" containerName="cleanup"
Nov 03 12:47:05.245318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244913    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd6aa02-3fd1-4aa9-b6e1-3555526ee59d" containerName="cleanup"
Nov 03 12:47:05.245318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244928    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f0239a-d786-4bad-a01d-9cb637ffdcd0" containerName="cleanup"
Nov 03 12:47:05.245318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244942    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy"
Nov 03 12:47:05.245318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.244957    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy"
Nov 03 12:47:05.249603 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.249585    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.344626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.344594    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-aa20ad-openshift-default-dockercfg-jszfj\""
Nov 03 12:47:05.349290 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.349261    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"]
Nov 03 12:47:05.358830 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.358801    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrsf\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-kube-api-access-5hrsf\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359130 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359097    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-podinfo\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359143    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-credential-socket\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359197    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d902052-bb91-4ca5-a9e4-0175332c1ead-istiod-ca-cert\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359283 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359278    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-socket\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359314    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-envoy\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359369    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-token\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359395    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-certs\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.359549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.359540    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-data\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461026    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-podinfo\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461063    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-credential-socket\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461113    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d902052-bb91-4ca5-a9e4-0175332c1ead-istiod-ca-cert\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461183    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-socket\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461242    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-envoy\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461521 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461333    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-token\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461533    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-credential-socket\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461643 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461366    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-certs\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461643 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461621    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-socket\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461751    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-certs\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461764    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-data\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.461905 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.461809    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrsf\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-kube-api-access-5hrsf\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.462079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.462054    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d902052-bb91-4ca5-a9e4-0175332c1ead-istiod-ca-cert\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.462185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.462061    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-data\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.463727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.463669    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-envoy\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.463950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.463925    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-podinfo\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.519028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.518570    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-token\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.525585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.525523    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrsf\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-kube-api-access-5hrsf\") pod \"e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") " pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:05.564966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:05.564579    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:06.005175 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.003724    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"]
Nov 03 12:47:06.007305 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:47:06.007281    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d902052_bb91_4ca5_a9e4_0175332c1ead.slice/crio-1106f7d267f6bd3c3b83ff5cff1bbd0f582e54bb9833734d0bfe943d4d2e6efa WatchSource:0}: Error finding container 1106f7d267f6bd3c3b83ff5cff1bbd0f582e54bb9833734d0bfe943d4d2e6efa: Status 404 returned error can't find the container with id 1106f7d267f6bd3c3b83ff5cff1bbd0f582e54bb9833734d0bfe943d4d2e6efa
Nov 03 12:47:06.010549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.010519    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:06.010633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.010583    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:06.010633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.010612    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:06.095483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.095411    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": read tcp 10.132.0.2:35998->10.132.0.167:15021: read: connection reset by peer" start-of-body=
Nov 03 12:47:06.095748 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.095679    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": read tcp 10.132.0.2:35998->10.132.0.167:15021: read: connection reset by peer"
Nov 03 12:47:06.129769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.129733    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" event={"ID":"7d902052-bb91-4ca5-a9e4-0175332c1ead","Type":"ContainerStarted","Data":"0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d"}
Nov 03 12:47:06.129899 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.129796    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" event={"ID":"7d902052-bb91-4ca5-a9e4-0175332c1ead","Type":"ContainerStarted","Data":"1106f7d267f6bd3c3b83ff5cff1bbd0f582e54bb9833734d0bfe943d4d2e6efa"}
Nov 03 12:47:06.239954 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.239860    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podStartSLOduration=1.239846603 podStartE2EDuration="1.239846603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:06.237199404 +0000 UTC m=+3375.249666081" watchObservedRunningTime="2026-11-03 12:47:06.239846603 +0000 UTC m=+3375.252313265"
Nov 03 12:47:06.565817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:06.565766    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:07.136927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:07.136898    4909 generic.go:356] "Generic (PLEG): container finished" podID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerID="80c0adcde17e37b6a184fe8633239529d2818b9938c3544e38901513ef078a60" exitCode=0
Nov 03 12:47:07.137096 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:07.136971    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" event={"ID":"679a8bb6-6ac5-455b-a303-0f6201ba1556","Type":"ContainerDied","Data":"80c0adcde17e37b6a184fe8633239529d2818b9938c3544e38901513ef078a60"}
Nov 03 12:47:07.137360 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:07.137342    4909 scope.go:122] "RemoveContainer" containerID="80c0adcde17e37b6a184fe8633239529d2818b9938c3544e38901513ef078a60"
Nov 03 12:47:07.138873 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:07.138846    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:07.138955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:07.138910    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:07.138955 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:07.138943    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:07.571313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:07.571281    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:08.166769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:08.166733    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" event={"ID":"679a8bb6-6ac5-455b-a303-0f6201ba1556","Type":"ContainerStarted","Data":"66dd79dedc499d33bc264fdcd3accf2b11e33a754129aadc2fae5ff22fbe6666"}
Nov 03 12:47:08.167024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:08.167003    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:08.168004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:08.167983    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:47:08.533499 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:08.533469    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:47:08.533684 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:08.533652    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 12:47:08.533787 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:08.533723    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 12:47:09.532929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:09.532893    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 12:47:09.533322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:09.532951    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 12:47:09.588436 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:47:09.588389    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:47:10.532929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:10.532892    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 12:47:10.533421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:10.532950    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 12:47:11.533915 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:11.533876    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 12:47:11.534421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:11.533924    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 12:47:12.530196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.530160    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"]
Nov 03 12:47:12.530849 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.530829    4909 cpu_manager.go:414] "containerMap: removing container" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy"
Nov 03 12:47:12.530978 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.530850    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04708f3d-162a-4077-af9d-ab6d62b12040" containerName="istio-proxy"
Nov 03 12:47:12.533320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.533293    4909 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 12:47:12.533435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.533353    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 12:47:12.533891 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.533873    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:12.629389 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.629336    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-aa20ad\""
Nov 03 12:47:12.629869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.629496    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-aa20ad-dockercfg-h4j7k\""
Nov 03 12:47:12.633569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.633538    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"]
Nov 03 12:47:12.710719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.709814    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmck\" (UniqueName: \"kubernetes.io/projected/62e94055-ab84-4679-b30f-4ce5c0ee93b8-kube-api-access-bdmck\") pod \"payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:12.710719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.709915    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e94055-ab84-4679-b30f-4ce5c0ee93b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:12.811468 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.811368    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmck\" (UniqueName: \"kubernetes.io/projected/62e94055-ab84-4679-b30f-4ce5c0ee93b8-kube-api-access-bdmck\") pod \"payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:12.811610 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.811522    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e94055-ab84-4679-b30f-4ce5c0ee93b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:12.812259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.812239    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e94055-ab84-4679-b30f-4ce5c0ee93b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:12.869778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:12.869679    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmck\" (UniqueName: \"kubernetes.io/projected/62e94055-ab84-4679-b30f-4ce5c0ee93b8-kube-api-access-bdmck\") pod \"payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:13.165225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:13.165138    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:13.537844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:13.537818    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:47:13.538257 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:13.538236    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:47:13.538916 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:13.538857    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"]
Nov 03 12:47:13.539031 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:13.539000    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:47:13.540597 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:47:13.540575    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e94055_ab84_4679_b30f_4ce5c0ee93b8.slice/crio-8bc5d9c429a75d8fe3ec933b8cba65796abc6f25ed45b68a60b7617d535f9c81 WatchSource:0}: Error finding container 8bc5d9c429a75d8fe3ec933b8cba65796abc6f25ed45b68a60b7617d535f9c81: Status 404 returned error can't find the container with id 8bc5d9c429a75d8fe3ec933b8cba65796abc6f25ed45b68a60b7617d535f9c81
Nov 03 12:47:14.208774 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:14.208736    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c" event={"ID":"62e94055-ab84-4679-b30f-4ce5c0ee93b8","Type":"ContainerStarted","Data":"b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d"}
Nov 03 12:47:14.209192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:14.208797    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c" event={"ID":"62e94055-ab84-4679-b30f-4ce5c0ee93b8","Type":"ContainerStarted","Data":"8bc5d9c429a75d8fe3ec933b8cba65796abc6f25ed45b68a60b7617d535f9c81"}
Nov 03 12:47:14.209192 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:14.209099    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:14.317320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:14.317270    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c" podStartSLOduration=2.31725686 podStartE2EDuration="2.31725686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:14.314171893 +0000 UTC m=+3383.326638556" watchObservedRunningTime="2026-11-03 12:47:14.31725686 +0000 UTC m=+3383.329723522"
Nov 03 12:47:17.130366 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:17.130333    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:47:23.345842 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.345810    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"]
Nov 03 12:47:23.351427 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.351403    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.411504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.411473    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"]
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460507    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283fb1be-e666-43df-b3c2-8dcbc288ec06-istiod-ca-cert\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460611    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-envoy\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460743    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-podinfo\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460806    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-socket\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460835    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c69g\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-kube-api-access-2c69g\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460868    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-data\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460908    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-credential-socket\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.460971    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-token\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.461007    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-certs\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.469720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.466837    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c84ae1-openshift-default-dockercfg-n96zl\""
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.562426    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-podinfo\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.562734    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-socket\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.562775    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c69g\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-kube-api-access-2c69g\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.562807    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-data\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.562846    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-credential-socket\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.562906    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-token\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.562941    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-certs\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.563015    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283fb1be-e666-43df-b3c2-8dcbc288ec06-istiod-ca-cert\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.563089    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-envoy\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.564480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.564290    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-data\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.565199 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.565055    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-socket\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.566505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.565615    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-credential-socket\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.566505 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.565893    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-certs\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.566721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.566510    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283fb1be-e666-43df-b3c2-8dcbc288ec06-istiod-ca-cert\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.567153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.567132    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-envoy\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.568371 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.568347    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-podinfo\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.618574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.618507    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-token\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.619831 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.619801    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c69g\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-kube-api-access-2c69g\") pod \"e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") " pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:23.666726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:23.665528    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:24.084820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.084790    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"]
Nov 03 12:47:24.122330 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:47:24.122305    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283fb1be_e666_43df_b3c2_8dcbc288ec06.slice/crio-13e871a6368c3adc23d40668fc2168e2528b64f056012fda0f19ab0c1e39f220 WatchSource:0}: Error finding container 13e871a6368c3adc23d40668fc2168e2528b64f056012fda0f19ab0c1e39f220: Status 404 returned error can't find the container with id 13e871a6368c3adc23d40668fc2168e2528b64f056012fda0f19ab0c1e39f220
Nov 03 12:47:24.124982 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.124950    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:24.125102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.125032    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:24.125102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.125078    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:47:24.269047 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.269013    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" event={"ID":"283fb1be-e666-43df-b3c2-8dcbc288ec06","Type":"ContainerStarted","Data":"b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda"}
Nov 03 12:47:24.269224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.269069    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" event={"ID":"283fb1be-e666-43df-b3c2-8dcbc288ec06","Type":"ContainerStarted","Data":"13e871a6368c3adc23d40668fc2168e2528b64f056012fda0f19ab0c1e39f220"}
Nov 03 12:47:24.381362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.381255    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podStartSLOduration=1.381235908 podStartE2EDuration="1.381235908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:24.377105623 +0000 UTC m=+3393.389572346" watchObservedRunningTime="2026-11-03 12:47:24.381235908 +0000 UTC m=+3393.393702572"
Nov 03 12:47:24.668563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:24.668491    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:25.219026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:25.218520    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:47:25.674430 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:25.674357    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:26.281459 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:26.281420    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:26.282457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:26.282436    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:47:39.564386 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:47:39.564342    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:47:46.415820 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.415789    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"]
Nov 03 12:47:46.422011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.421988    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.472846 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.472823    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"]
Nov 03 12:47:46.485597 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.485566    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28854ff1-e49b-4aac-9b7a-386f471b01b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.485799 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.485778    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnxs\" (UniqueName: \"kubernetes.io/projected/28854ff1-e49b-4aac-9b7a-386f471b01b0-kube-api-access-spnxs\") pod \"payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.515924 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.515903    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-38f178\""
Nov 03 12:47:46.516024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.515945    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-38f178-dockercfg-mg4p9\""
Nov 03 12:47:46.580408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.580382    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"]
Nov 03 12:47:46.584874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.584853    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:46.586657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.586628    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28854ff1-e49b-4aac-9b7a-386f471b01b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.586791 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.586740    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnxs\" (UniqueName: \"kubernetes.io/projected/28854ff1-e49b-4aac-9b7a-386f471b01b0-kube-api-access-spnxs\") pod \"payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.587551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.587528    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28854ff1-e49b-4aac-9b7a-386f471b01b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.648960 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.648926    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"]
Nov 03 12:47:46.659476 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.659439    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnxs\" (UniqueName: \"kubernetes.io/projected/28854ff1-e49b-4aac-9b7a-386f471b01b0-kube-api-access-spnxs\") pod \"payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.718278 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.715617    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pq7\" (UniqueName: \"kubernetes.io/projected/1d0faaca-e103-423a-aa1c-d646d5600033-kube-api-access-45pq7\") pod \"payload-processing-e2e-extmod-a-38f178-54844c864f-847dk\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:46.718278 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.715825    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0faaca-e103-423a-aa1c-d646d5600033-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-38f178-54844c864f-847dk\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:46.736903 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.736874    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:46.821738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.816989    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pq7\" (UniqueName: \"kubernetes.io/projected/1d0faaca-e103-423a-aa1c-d646d5600033-kube-api-access-45pq7\") pod \"payload-processing-e2e-extmod-a-38f178-54844c864f-847dk\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:46.821738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.817236    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0faaca-e103-423a-aa1c-d646d5600033-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-38f178-54844c864f-847dk\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:46.821738 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.818948    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0faaca-e103-423a-aa1c-d646d5600033-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-38f178-54844c864f-847dk\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:46.870508 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.870476    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pq7\" (UniqueName: \"kubernetes.io/projected/1d0faaca-e103-423a-aa1c-d646d5600033-kube-api-access-45pq7\") pod \"payload-processing-e2e-extmod-a-38f178-54844c864f-847dk\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:46.902862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:46.902826    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:47.142022 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.141990    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"]
Nov 03 12:47:47.143200 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:47:47.143176    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28854ff1_e49b_4aac_9b7a_386f471b01b0.slice/crio-ec7773b84e2e3634d6cd3cb3afaddbf32915d0651ea7a329dc79daeb0602217c WatchSource:0}: Error finding container ec7773b84e2e3634d6cd3cb3afaddbf32915d0651ea7a329dc79daeb0602217c: Status 404 returned error can't find the container with id ec7773b84e2e3634d6cd3cb3afaddbf32915d0651ea7a329dc79daeb0602217c
Nov 03 12:47:47.283293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.283269    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"]
Nov 03 12:47:47.284294 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:47:47.284270    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0faaca_e103_423a_aa1c_d646d5600033.slice/crio-760a265d5cc88f4dda74f46d9399f5ad8486ef61d42b87e324e01500cae242bc WatchSource:0}: Error finding container 760a265d5cc88f4dda74f46d9399f5ad8486ef61d42b87e324e01500cae242bc: Status 404 returned error can't find the container with id 760a265d5cc88f4dda74f46d9399f5ad8486ef61d42b87e324e01500cae242bc
Nov 03 12:47:47.413876 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.413837    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv" event={"ID":"28854ff1-e49b-4aac-9b7a-386f471b01b0","Type":"ContainerStarted","Data":"82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e"}
Nov 03 12:47:47.414008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.413910    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:47.414008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.413935    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv" event={"ID":"28854ff1-e49b-4aac-9b7a-386f471b01b0","Type":"ContainerStarted","Data":"ec7773b84e2e3634d6cd3cb3afaddbf32915d0651ea7a329dc79daeb0602217c"}
Nov 03 12:47:47.415726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.415665    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk" event={"ID":"1d0faaca-e103-423a-aa1c-d646d5600033","Type":"ContainerStarted","Data":"439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978"}
Nov 03 12:47:47.415844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.415738    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk" event={"ID":"1d0faaca-e103-423a-aa1c-d646d5600033","Type":"ContainerStarted","Data":"760a265d5cc88f4dda74f46d9399f5ad8486ef61d42b87e324e01500cae242bc"}
Nov 03 12:47:47.415844 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.415761    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:47:47.520219 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.520171    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv" podStartSLOduration=1.520158001 podStartE2EDuration="1.520158001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:47.518407341 +0000 UTC m=+3416.530874004" watchObservedRunningTime="2026-11-03 12:47:47.520158001 +0000 UTC m=+3416.532624664"
Nov 03 12:47:47.627373 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:47.626020    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk" podStartSLOduration=1.6260048980000001 podStartE2EDuration="1.626004898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:47.622899061 +0000 UTC m=+3416.635365736" watchObservedRunningTime="2026-11-03 12:47:47.626004898 +0000 UTC m=+3416.638471560"
Nov 03 12:47:58.425331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:58.425299    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:47:58.425772 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:47:58.425375    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:48:01.756226 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:01.755452    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": read tcp 10.132.0.2:47922->10.132.0.170:15021: read: connection reset by peer" start-of-body=
Nov 03 12:48:01.756226 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:01.755521    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": read tcp 10.132.0.2:47922->10.132.0.170:15021: read: connection reset by peer"
Nov 03 12:48:02.527626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:02.527595    4909 generic.go:356] "Generic (PLEG): container finished" podID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerID="963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94" exitCode=0
Nov 03 12:48:02.527845 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:02.527670    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" event={"ID":"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6","Type":"ContainerDied","Data":"963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94"}
Nov 03 12:48:02.528100 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:02.528083    4909 scope.go:122] "RemoveContainer" containerID="963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94"
Nov 03 12:48:02.529601 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:02.529571    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:48:02.529682 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:02.529643    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:48:02.529682 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:02.529675    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:48:03.536086 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:03.536051    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" event={"ID":"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6","Type":"ContainerStarted","Data":"8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661"}
Nov 03 12:48:04.009422 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:04.009389    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:48:04.009721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:04.009675    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 12:48:04.009784 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:04.009756    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 12:48:05.009395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:05.009362    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 12:48:05.009874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:05.009413    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 12:48:06.009571 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:06.009538    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 12:48:06.010082 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:06.009603    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 12:48:07.009583 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:07.009548    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 12:48:07.010012 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:07.009613    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 12:48:08.009322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:08.009287    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 12:48:08.009511 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:08.009343    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 12:48:09.013590 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:09.013560    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:48:09.014046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:09.013980    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:48:09.014577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:09.014559    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 12:48:09.564927 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:48:09.564893    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:48:20.844653 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:48:20.844617    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:48:39.564318 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:48:39.564239    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:49:00.930854 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:00.930823    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:49:04.155414 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.155380    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"]
Nov 03 12:49:04.159694 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.159672    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.255404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.255339    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"kube-root-ca.crt\""
Nov 03 12:49:04.255404 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.255398    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"openshift-service-ca.crt\""
Nov 03 12:49:04.255658 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.255471    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"auth-test-model-6fb3a8-kserve-self-signed-certs\""
Nov 03 12:49:04.255658 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.255480    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"default-dockercfg-zlm8p\""
Nov 03 12:49:04.258770 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.258748    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"]
Nov 03 12:49:04.333795 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.333755    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-dshm\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.333977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.333810    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tls-certs\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.333977 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.333917    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-model-cache\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.334068 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.333992    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5994\" (UniqueName: \"kubernetes.io/projected/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-kube-api-access-t5994\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.334105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.334084    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-home\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.334149 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.334118    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tmp-dir\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435372    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-model-cache\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435481    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5994\" (UniqueName: \"kubernetes.io/projected/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-kube-api-access-t5994\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435545    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-home\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435592    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tmp-dir\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435686    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-dshm\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435764    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tls-certs\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435793    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-model-cache\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435878    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-home\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.435998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.435987    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tmp-dir\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.437998 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.437977    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-dshm\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.438429 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.438405    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tls-certs\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.487691 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.487657    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5994\" (UniqueName: \"kubernetes.io/projected/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-kube-api-access-t5994\") pod \"auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:04.772794 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:04.772763    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:05.139495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:05.139465    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"]
Nov 03 12:49:05.140887 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:49:05.140864    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79de6af0_d2d4_43d2_97e7_d8c8d70bf6d6.slice/crio-109fdb8702371231333c621b14159aa3ef989c1d4094e5bfdea6688a3fe6e4e8 WatchSource:0}: Error finding container 109fdb8702371231333c621b14159aa3ef989c1d4094e5bfdea6688a3fe6e4e8: Status 404 returned error can't find the container with id 109fdb8702371231333c621b14159aa3ef989c1d4094e5bfdea6688a3fe6e4e8
Nov 03 12:49:05.988573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:05.988533    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw" event={"ID":"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6","Type":"ContainerStarted","Data":"1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9"}
Nov 03 12:49:05.988968 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:05.988584    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw" event={"ID":"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6","Type":"ContainerStarted","Data":"109fdb8702371231333c621b14159aa3ef989c1d4094e5bfdea6688a3fe6e4e8"}
Nov 03 12:49:06.093574 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:06.093519    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw" podStartSLOduration=2.093505741 podStartE2EDuration="2.093505741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:49:06.092575048 +0000 UTC m=+3495.105041723" watchObservedRunningTime="2026-11-03 12:49:06.093505741 +0000 UTC m=+3495.105972438"
Nov 03 12:49:06.773568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:06.773534    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:09.564082 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:49:09.564047    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:49:14.773665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:14.773624    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:14.786378 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:14.786356    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:15.057848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:15.057768    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:49:33.165752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:33.165720    4909 generic.go:356] "Generic (PLEG): container finished" podID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerID="b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda" exitCode=0
Nov 03 12:49:33.165752 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:33.165736    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" event={"ID":"283fb1be-e666-43df-b3c2-8dcbc288ec06","Type":"ContainerDied","Data":"b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda"}
Nov 03 12:49:33.166222 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:33.166050    4909 scope.go:122] "RemoveContainer" containerID="b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda"
Nov 03 12:49:33.167618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:33.167590    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:49:33.167840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:33.167654    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:49:33.167840 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:33.167685    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:49:34.174934 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:34.174899    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" event={"ID":"283fb1be-e666-43df-b3c2-8dcbc288ec06","Type":"ContainerStarted","Data":"13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d"}
Nov 03 12:49:34.668850 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:34.668814    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:49:34.669041 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:34.669023    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 12:49:34.669090 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:34.669069    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 12:49:35.669156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:35.669123    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 12:49:35.669569 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:35.669187    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 12:49:36.669196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:36.669159    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 12:49:36.669617 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:36.669228    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 12:49:37.669343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:37.669307    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 12:49:37.669775 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:37.669375    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 12:49:38.672088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:38.672047    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 12:49:38.672537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:38.672110    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 12:49:39.564495 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:49:39.564463    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:49:39.673167 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:39.673136    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:49:39.673562 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:39.673475    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:49:39.674134 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:39.674116    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:49:50.485626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:49:50.485592    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:50:09.619858 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:50:09.619818    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:50:18.506046 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:18.506015    4909 generic.go:356] "Generic (PLEG): container finished" podID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerID="0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d" exitCode=0
Nov 03 12:50:18.506501 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:18.506051    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" event={"ID":"7d902052-bb91-4ca5-a9e4-0175332c1ead","Type":"ContainerDied","Data":"0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d"}
Nov 03 12:50:18.506564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:18.506552    4909 scope.go:122] "RemoveContainer" containerID="0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d"
Nov 03 12:50:18.508351 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:18.508308    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:50:18.508464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:18.508375    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:50:18.508464 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:18.508420    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:50:19.514593 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:19.514558    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" event={"ID":"7d902052-bb91-4ca5-a9e4-0175332c1ead","Type":"ContainerStarted","Data":"35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f"}
Nov 03 12:50:19.565497 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:19.565456    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:50:19.566953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:19.566929    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 03 12:50:19.567022 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:19.566978    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 03 12:50:20.566208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:20.566169    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 03 12:50:20.566624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:20.566222    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 03 12:50:21.565781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:21.565745    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 03 12:50:21.566048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:21.565807    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 03 12:50:22.566209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:22.566174    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 03 12:50:22.566614 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:22.566223    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 03 12:50:23.167035 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:23.167004    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:50:23.565310 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:23.565264    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 03 12:50:23.565506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:23.565356    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 03 12:50:24.565927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:24.565886    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 03 12:50:24.566320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:24.565960    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 03 12:50:25.571602 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:25.571573    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:50:25.572579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:25.572558    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:50:33.276723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:33.273230    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"]
Nov 03 12:50:38.274571 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.274531    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8" podUID="f1c830fa-1dcf-48a4-adbe-927013af88bb" containerName="payload-processing" containerID="cri-o://8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee" gracePeriod=25
Nov 03 12:50:38.555524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.555498    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:50:38.660240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.660209    4909 generic.go:356] "Generic (PLEG): container finished" podID="f1c830fa-1dcf-48a4-adbe-927013af88bb" containerID="8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee" exitCode=0
Nov 03 12:50:38.660409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.660262    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"
Nov 03 12:50:38.660409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.660292    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8" event={"ID":"f1c830fa-1dcf-48a4-adbe-927013af88bb","Type":"ContainerDied","Data":"8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee"}
Nov 03 12:50:38.660409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.660355    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8" event={"ID":"f1c830fa-1dcf-48a4-adbe-927013af88bb","Type":"ContainerDied","Data":"06bdf1915ba875d4cd1f721ad7aac21c19e82a929a53f5a7bac552318047b894"}
Nov 03 12:50:38.660409 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.660378    4909 scope.go:122] "RemoveContainer" containerID="8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee"
Nov 03 12:50:38.671312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.671288    4909 scope.go:122] "RemoveContainer" containerID="8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee"
Nov 03 12:50:38.671627 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:50:38.671604    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee\": container with ID starting with 8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee not found: ID does not exist" containerID="8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee"
Nov 03 12:50:38.671763 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.671634    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee"} err="failed to get container status \"8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee\": rpc error: code = NotFound desc = could not find container \"8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee\": container with ID starting with 8cce8f8bdc638fa3369f8b53e756053f43f566350d4496d8c90d5252fcc951ee not found: ID does not exist"
Nov 03 12:50:38.719452 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.719422    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c830fa-1dcf-48a4-adbe-927013af88bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c830fa-1dcf-48a4-adbe-927013af88bb-plugins-config-volume\") pod \"f1c830fa-1dcf-48a4-adbe-927013af88bb\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") "
Nov 03 12:50:38.719630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.719472    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c830fa-1dcf-48a4-adbe-927013af88bb-kube-api-access-mxlc6\" (UniqueName: \"kubernetes.io/projected/f1c830fa-1dcf-48a4-adbe-927013af88bb-kube-api-access-mxlc6\") pod \"f1c830fa-1dcf-48a4-adbe-927013af88bb\" (UID: \"f1c830fa-1dcf-48a4-adbe-927013af88bb\") "
Nov 03 12:50:38.719943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.719916    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c830fa-1dcf-48a4-adbe-927013af88bb-plugins-config-volume" pod "f1c830fa-1dcf-48a4-adbe-927013af88bb" (UID: "f1c830fa-1dcf-48a4-adbe-927013af88bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:38.720329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.720302    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c830fa-1dcf-48a4-adbe-927013af88bb-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:38.722080 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.722051    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c830fa-1dcf-48a4-adbe-927013af88bb-kube-api-access-mxlc6" pod "f1c830fa-1dcf-48a4-adbe-927013af88bb" (UID: "f1c830fa-1dcf-48a4-adbe-927013af88bb"). InnerVolumeSpecName "kube-api-access-mxlc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:38.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:38.821104    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxlc6\" (UniqueName: \"kubernetes.io/projected/f1c830fa-1dcf-48a4-adbe-927013af88bb-kube-api-access-mxlc6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:39.073538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:39.073449    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"]
Nov 03 12:50:39.076335 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:39.076311    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fd7881-6dd776dcc9-tkmw8"]
Nov 03 12:50:39.129344 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:39.129306    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c830fa-1dcf-48a4-adbe-927013af88bb" path="/var/lib/kubelet/pods/f1c830fa-1dcf-48a4-adbe-927013af88bb/volumes"
Nov 03 12:50:39.564528 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:50:39.564495    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:50:47.015012 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:47.014980    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"]
Nov 03 12:50:47.125045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:47.125013    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"]
Nov 03 12:50:47.930928 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:47.930895    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:50:51.588120 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:51.588090    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:50:51.595813 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:51.595791    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:50:52.016269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.016228    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk" podUID="1d0faaca-e103-423a-aa1c-d646d5600033" containerName="payload-processing" containerID="cri-o://439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978" gracePeriod=25
Nov 03 12:50:52.126330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.126295    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv" podUID="28854ff1-e49b-4aac-9b7a-386f471b01b0" containerName="payload-pre-processing" containerID="cri-o://82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e" gracePeriod=25
Nov 03 12:50:52.302436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.302406    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:50:52.396215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.396185    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0faaca-e103-423a-aa1c-d646d5600033-kube-api-access-45pq7\" (UniqueName: \"kubernetes.io/projected/1d0faaca-e103-423a-aa1c-d646d5600033-kube-api-access-45pq7\") pod \"1d0faaca-e103-423a-aa1c-d646d5600033\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") "
Nov 03 12:50:52.396382 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.396366    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0faaca-e103-423a-aa1c-d646d5600033-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0faaca-e103-423a-aa1c-d646d5600033-plugins-config-volume\") pod \"1d0faaca-e103-423a-aa1c-d646d5600033\" (UID: \"1d0faaca-e103-423a-aa1c-d646d5600033\") "
Nov 03 12:50:52.396764 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.396739    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0faaca-e103-423a-aa1c-d646d5600033-plugins-config-volume" pod "1d0faaca-e103-423a-aa1c-d646d5600033" (UID: "1d0faaca-e103-423a-aa1c-d646d5600033"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:52.396994 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.396973    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0faaca-e103-423a-aa1c-d646d5600033-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:52.398295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.398271    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0faaca-e103-423a-aa1c-d646d5600033-kube-api-access-45pq7" pod "1d0faaca-e103-423a-aa1c-d646d5600033" (UID: "1d0faaca-e103-423a-aa1c-d646d5600033"). InnerVolumeSpecName "kube-api-access-45pq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:52.405225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.405207    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:50:52.497514 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.497484    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28854ff1-e49b-4aac-9b7a-386f471b01b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28854ff1-e49b-4aac-9b7a-386f471b01b0-plugins-config-volume\") pod \"28854ff1-e49b-4aac-9b7a-386f471b01b0\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") "
Nov 03 12:50:52.497657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.497543    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28854ff1-e49b-4aac-9b7a-386f471b01b0-kube-api-access-spnxs\" (UniqueName: \"kubernetes.io/projected/28854ff1-e49b-4aac-9b7a-386f471b01b0-kube-api-access-spnxs\") pod \"28854ff1-e49b-4aac-9b7a-386f471b01b0\" (UID: \"28854ff1-e49b-4aac-9b7a-386f471b01b0\") "
Nov 03 12:50:52.497929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.497904    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28854ff1-e49b-4aac-9b7a-386f471b01b0-plugins-config-volume" pod "28854ff1-e49b-4aac-9b7a-386f471b01b0" (UID: "28854ff1-e49b-4aac-9b7a-386f471b01b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:52.498108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.498092    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28854ff1-e49b-4aac-9b7a-386f471b01b0-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:52.498151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.498110    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45pq7\" (UniqueName: \"kubernetes.io/projected/1d0faaca-e103-423a-aa1c-d646d5600033-kube-api-access-45pq7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:52.499766 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.499740    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28854ff1-e49b-4aac-9b7a-386f471b01b0-kube-api-access-spnxs" pod "28854ff1-e49b-4aac-9b7a-386f471b01b0" (UID: "28854ff1-e49b-4aac-9b7a-386f471b01b0"). InnerVolumeSpecName "kube-api-access-spnxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:52.599250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.599179    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnxs\" (UniqueName: \"kubernetes.io/projected/28854ff1-e49b-4aac-9b7a-386f471b01b0-kube-api-access-spnxs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:52.779003 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.778971    4909 generic.go:356] "Generic (PLEG): container finished" podID="28854ff1-e49b-4aac-9b7a-386f471b01b0" containerID="82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e" exitCode=0
Nov 03 12:50:52.779180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.779017    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"
Nov 03 12:50:52.779180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.779056    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv" event={"ID":"28854ff1-e49b-4aac-9b7a-386f471b01b0","Type":"ContainerDied","Data":"82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e"}
Nov 03 12:50:52.779180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.779126    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv" event={"ID":"28854ff1-e49b-4aac-9b7a-386f471b01b0","Type":"ContainerDied","Data":"ec7773b84e2e3634d6cd3cb3afaddbf32915d0651ea7a329dc79daeb0602217c"}
Nov 03 12:50:52.779180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.779157    4909 scope.go:122] "RemoveContainer" containerID="82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e"
Nov 03 12:50:52.780665 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.780642    4909 generic.go:356] "Generic (PLEG): container finished" podID="1d0faaca-e103-423a-aa1c-d646d5600033" containerID="439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978" exitCode=0
Nov 03 12:50:52.780809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.780673    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk" event={"ID":"1d0faaca-e103-423a-aa1c-d646d5600033","Type":"ContainerDied","Data":"439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978"}
Nov 03 12:50:52.780809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.780687    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"
Nov 03 12:50:52.780809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.780753    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk" event={"ID":"1d0faaca-e103-423a-aa1c-d646d5600033","Type":"ContainerDied","Data":"760a265d5cc88f4dda74f46d9399f5ad8486ef61d42b87e324e01500cae242bc"}
Nov 03 12:50:52.791449 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.791435    4909 scope.go:122] "RemoveContainer" containerID="82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e"
Nov 03 12:50:52.791668 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:50:52.791649    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e\": container with ID starting with 82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e not found: ID does not exist" containerID="82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e"
Nov 03 12:50:52.791781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.791679    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e"} err="failed to get container status \"82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e\": rpc error: code = NotFound desc = could not find container \"82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e\": container with ID starting with 82f862dce7defbdb1b45ec45ab9bb8176e76e536a48dc586d1dfaf77170df02e not found: ID does not exist"
Nov 03 12:50:52.791781 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.791728    4909 scope.go:122] "RemoveContainer" containerID="439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978"
Nov 03 12:50:52.800353 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.800336    4909 scope.go:122] "RemoveContainer" containerID="439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978"
Nov 03 12:50:52.800597 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:50:52.800579    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978\": container with ID starting with 439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978 not found: ID does not exist" containerID="439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978"
Nov 03 12:50:52.800655 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.800604    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978"} err="failed to get container status \"439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978\": rpc error: code = NotFound desc = could not find container \"439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978\": container with ID starting with 439c098da5c391f6aaccfcd7f51f77cbb04982bfd34fac073e7c564461e02978 not found: ID does not exist"
Nov 03 12:50:52.931102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.931043    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"]
Nov 03 12:50:52.977851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:52.977823    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-38f178-59f6c57ff5-dxvlv"]
Nov 03 12:50:53.079436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:53.079404    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"]
Nov 03 12:50:53.084220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:53.084192    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-38f178-54844c864f-847dk"]
Nov 03 12:50:53.125145 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:53.125111    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0faaca-e103-423a-aa1c-d646d5600033" path="/var/lib/kubelet/pods/1d0faaca-e103-423a-aa1c-d646d5600033/volumes"
Nov 03 12:50:53.125587 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:50:53.125566    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28854ff1-e49b-4aac-9b7a-386f471b01b0" path="/var/lib/kubelet/pods/28854ff1-e49b-4aac-9b7a-386f471b01b0/volumes"
Nov 03 12:51:09.564476 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:51:09.564440    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:51:35.946453 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:35.946377    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"]
Nov 03 12:51:35.946995 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:35.946596    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy" containerID="cri-o://8a9c41c590ca9acb7aeb3cc7b2f05763b2ff228bb69643a9522199bae9a97780" gracePeriod=30
Nov 03 12:51:39.563890 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:51:39.563852    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:51:41.150568 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.150532    4909 generic.go:356] "Generic (PLEG): container finished" podID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerID="8a9c41c590ca9acb7aeb3cc7b2f05763b2ff228bb69643a9522199bae9a97780" exitCode=0
Nov 03 12:51:41.151153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.150738    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" event={"ID":"a01fbd75-25d5-4efd-8788-44a86c84cf1f","Type":"ContainerDied","Data":"8a9c41c590ca9acb7aeb3cc7b2f05763b2ff228bb69643a9522199bae9a97780"}
Nov 03 12:51:41.151153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.150798    4909 scope.go:122] "RemoveContainer" containerID="705fe81b0fb806238ac92568283ac90ba5006f9a202d1a61161383daa5b61f89"
Nov 03 12:51:41.230548 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.230525    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:51:41.274877 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.274853    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-podinfo\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.274908    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-certs\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275019 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.274974    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-kube-api-access-n5sk4\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-kube-api-access-n5sk4\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275040    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-token\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-token\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275100    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istiod-ca-cert\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275150 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275137    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-credential-socket\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275173    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-envoy\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275203    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-certs" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:41.275424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275313    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-data\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275380    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-socket\") pod \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\" (UID: \"a01fbd75-25d5-4efd-8788-44a86c84cf1f\") "
Nov 03 12:51:41.275424 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275405    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-credential-socket" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:41.275577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275452    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istiod-ca-cert" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:51:41.275667 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275646    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-data" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:41.275837 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275812    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-socket" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:41.275922 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275905    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.275987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275929    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.275987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275944    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.275987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275958    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.275987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.275973    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.277299 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.277278    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-envoy" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:41.277481 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.277466    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-token" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:41.277618 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.277595    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-podinfo" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:51:41.277727 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.277634    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-kube-api-access-n5sk4" pod "a01fbd75-25d5-4efd-8788-44a86c84cf1f" (UID: "a01fbd75-25d5-4efd-8788-44a86c84cf1f"). InnerVolumeSpecName "kube-api-access-n5sk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:41.376598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.376568    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.376598 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.376594    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.376787 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.376604    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5sk4\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-kube-api-access-n5sk4\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:41.376787 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:41.376614    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01fbd75-25d5-4efd-8788-44a86c84cf1f-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:42.159249 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:42.159214    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"
Nov 03 12:51:42.159694 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:42.159267    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr" event={"ID":"a01fbd75-25d5-4efd-8788-44a86c84cf1f","Type":"ContainerDied","Data":"d34d5cbbd7d46149a04c25281ac762dce23fa2009376f2fde11f59b27f175377"}
Nov 03 12:51:42.159694 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:42.159319    4909 scope.go:122] "RemoveContainer" containerID="8a9c41c590ca9acb7aeb3cc7b2f05763b2ff228bb69643a9522199bae9a97780"
Nov 03 12:51:42.357399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:42.357368    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"]
Nov 03 12:51:42.403001 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:42.402974    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bebc6d-openshift-default-85f9fbc57-tkcrr"]
Nov 03 12:51:43.123852 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:43.123820    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" path="/var/lib/kubelet/pods/a01fbd75-25d5-4efd-8788-44a86c84cf1f/volumes"
Nov 03 12:51:43.569442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:43.569409    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"]
Nov 03 12:51:44.567408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:44.567369    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:51:48.570182 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.570145    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c" podUID="62e94055-ab84-4679-b30f-4ce5c0ee93b8" containerName="payload-pre-processing" containerID="cri-o://b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d" gracePeriod=25
Nov 03 12:51:48.828209 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.828155    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:51:48.868619 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.868591    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e94055-ab84-4679-b30f-4ce5c0ee93b8-kube-api-access-bdmck\" (UniqueName: \"kubernetes.io/projected/62e94055-ab84-4679-b30f-4ce5c0ee93b8-kube-api-access-bdmck\") pod \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") "
Nov 03 12:51:48.868800 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.868774    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e94055-ab84-4679-b30f-4ce5c0ee93b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e94055-ab84-4679-b30f-4ce5c0ee93b8-plugins-config-volume\") pod \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\" (UID: \"62e94055-ab84-4679-b30f-4ce5c0ee93b8\") "
Nov 03 12:51:48.869205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.869176    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e94055-ab84-4679-b30f-4ce5c0ee93b8-plugins-config-volume" pod "62e94055-ab84-4679-b30f-4ce5c0ee93b8" (UID: "62e94055-ab84-4679-b30f-4ce5c0ee93b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:51:48.870632 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.870607    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e94055-ab84-4679-b30f-4ce5c0ee93b8-kube-api-access-bdmck" pod "62e94055-ab84-4679-b30f-4ce5c0ee93b8" (UID: "62e94055-ab84-4679-b30f-4ce5c0ee93b8"). InnerVolumeSpecName "kube-api-access-bdmck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:48.971004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.970971    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e94055-ab84-4679-b30f-4ce5c0ee93b8-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:48.971004 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:48.971008    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmck\" (UniqueName: \"kubernetes.io/projected/62e94055-ab84-4679-b30f-4ce5c0ee93b8-kube-api-access-bdmck\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:49.203818 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.203786    4909 generic.go:356] "Generic (PLEG): container finished" podID="62e94055-ab84-4679-b30f-4ce5c0ee93b8" containerID="b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d" exitCode=0
Nov 03 12:51:49.203954 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.203832    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"
Nov 03 12:51:49.203954 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.203841    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c" event={"ID":"62e94055-ab84-4679-b30f-4ce5c0ee93b8","Type":"ContainerDied","Data":"b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d"}
Nov 03 12:51:49.203954 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.203880    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c" event={"ID":"62e94055-ab84-4679-b30f-4ce5c0ee93b8","Type":"ContainerDied","Data":"8bc5d9c429a75d8fe3ec933b8cba65796abc6f25ed45b68a60b7617d535f9c81"}
Nov 03 12:51:49.203954 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.203900    4909 scope.go:122] "RemoveContainer" containerID="b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d"
Nov 03 12:51:49.213956 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.213936    4909 scope.go:122] "RemoveContainer" containerID="b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d"
Nov 03 12:51:49.214230 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:51:49.214205    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d\": container with ID starting with b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d not found: ID does not exist" containerID="b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d"
Nov 03 12:51:49.214284 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.214234    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d"} err="failed to get container status \"b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d\": rpc error: code = NotFound desc = could not find container \"b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d\": container with ID starting with b79cdf9170323c65fad509dce4ac9601429878ed7995ec819a08273d7236336d not found: ID does not exist"
Nov 03 12:51:49.355503 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.355478    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"]
Nov 03 12:51:49.358225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:49.358202    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-aa20ad-7d6799c44c-xg86c"]
Nov 03 12:51:51.136483 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:51:51.136445    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e94055-ab84-4679-b30f-4ce5c0ee93b8" path="/var/lib/kubelet/pods/62e94055-ab84-4679-b30f-4ce5c0ee93b8/volumes"
Nov 03 12:52:05.135658 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:05.135625    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"]
Nov 03 12:52:05.136119 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:05.135830    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy" containerID="cri-o://66dd79dedc499d33bc264fdcd3accf2b11e33a754129aadc2fae5ff22fbe6666" gracePeriod=30
Nov 03 12:52:09.563847 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:52:09.563810    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:52:10.352573 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.352540    4909 generic.go:356] "Generic (PLEG): container finished" podID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerID="66dd79dedc499d33bc264fdcd3accf2b11e33a754129aadc2fae5ff22fbe6666" exitCode=0
Nov 03 12:52:10.352793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.352609    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" event={"ID":"679a8bb6-6ac5-455b-a303-0f6201ba1556","Type":"ContainerDied","Data":"66dd79dedc499d33bc264fdcd3accf2b11e33a754129aadc2fae5ff22fbe6666"}
Nov 03 12:52:10.352793 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.352673    4909 scope.go:122] "RemoveContainer" containerID="80c0adcde17e37b6a184fe8633239529d2818b9938c3544e38901513ef078a60"
Nov 03 12:52:10.417436 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.417378    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:52:10.443101 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443069    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-podinfo\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443147    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-credential-socket\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443224 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443188    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-envoy\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443352 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443241    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-kube-api-access-sbswv\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-kube-api-access-sbswv\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443352 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443299    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-socket\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443372    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-token\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-token\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443424    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679a8bb6-6ac5-455b-a303-0f6201ba1556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679a8bb6-6ac5-455b-a303-0f6201ba1556-istiod-ca-cert\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443454    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-credential-socket" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:52:10.443640 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443462    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-data\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443640 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443535    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-certs\") pod \"679a8bb6-6ac5-455b-a303-0f6201ba1556\" (UID: \"679a8bb6-6ac5-455b-a303-0f6201ba1556\") "
Nov 03 12:52:10.443788 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443689    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-data" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.443924    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679a8bb6-6ac5-455b-a303-0f6201ba1556-istiod-ca-cert" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.444034    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-certs" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.444085    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-socket" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.444209    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.444225    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679a8bb6-6ac5-455b-a303-0f6201ba1556-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.444235    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.444244    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.444275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.444256    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.445391 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.445369    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-envoy" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:52:10.445540 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.445524    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-token" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:10.445777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.445753    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-podinfo" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:52:10.445777 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.445763    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-kube-api-access-sbswv" pod "679a8bb6-6ac5-455b-a303-0f6201ba1556" (UID: "679a8bb6-6ac5-455b-a303-0f6201ba1556"). InnerVolumeSpecName "kube-api-access-sbswv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:10.545078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.545048    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.545078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.545073    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.545078 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.545083    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbswv\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-kube-api-access-sbswv\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:10.545282 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:10.545094    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679a8bb6-6ac5-455b-a303-0f6201ba1556-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:11.360196 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:11.360167    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"
Nov 03 12:52:11.360613 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:11.360205    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc" event={"ID":"679a8bb6-6ac5-455b-a303-0f6201ba1556","Type":"ContainerDied","Data":"ca2aa12bfb4a7c644a764b80b86db4b9ede8772be165446cfe2b6bf6a344808f"}
Nov 03 12:52:11.360613 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:11.360263    4909 scope.go:122] "RemoveContainer" containerID="66dd79dedc499d33bc264fdcd3accf2b11e33a754129aadc2fae5ff22fbe6666"
Nov 03 12:52:11.512865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:11.512838    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"]
Nov 03 12:52:11.558715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:11.558679    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-62pgc"]
Nov 03 12:52:13.123834 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:13.123803    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" path="/var/lib/kubelet/pods/679a8bb6-6ac5-455b-a303-0f6201ba1556/volumes"
Nov 03 12:52:39.563586 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:52:39.563548    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:52:42.845380 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:52:42.845339    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:53:04.227204 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:04.227126    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"]
Nov 03 12:53:04.227644 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:04.227334    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy" containerID="cri-o://35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f" gracePeriod=30
Nov 03 12:53:09.492884 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.492860    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:53:09.515108 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515085    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d902052-bb91-4ca5-a9e4-0175332c1ead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d902052-bb91-4ca5-a9e4-0175332c1ead-istiod-ca-cert\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515130    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-socket\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515205 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515165    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-certs\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515317 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515276    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-credential-socket\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515441 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515425    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-data\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515404    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-certs" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:09.515551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515532    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-envoy\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515445    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d902052-bb91-4ca5-a9e4-0175332c1ead-istiod-ca-cert" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:09.515551 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515448    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-socket" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:09.515883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515563    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-token\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-token\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515581    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-credential-socket" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:09.515883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515603    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-podinfo\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.515883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515604    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-data" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:09.515883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.515635    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-kube-api-access-5hrsf\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-kube-api-access-5hrsf\") pod \"7d902052-bb91-4ca5-a9e4-0175332c1ead\" (UID: \"7d902052-bb91-4ca5-a9e4-0175332c1ead\") "
Nov 03 12:53:09.516176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.516107    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d902052-bb91-4ca5-a9e4-0175332c1ead-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.516176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.516127    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.516176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.516140    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.516176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.516154    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.516176 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.516164    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.517679 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.517654    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-token" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:09.517790 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.517771    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-podinfo" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:53:09.517988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.517967    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-envoy" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:09.518073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.518009    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-kube-api-access-5hrsf" pod "7d902052-bb91-4ca5-a9e4-0175332c1ead" (UID: "7d902052-bb91-4ca5-a9e4-0175332c1ead"). InnerVolumeSpecName "kube-api-access-5hrsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:09.564123 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:53:09.564093    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:53:09.617099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.617059    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.617099 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.617092    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hrsf\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-kube-api-access-5hrsf\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.617302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.617108    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.617302 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.617121    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d902052-bb91-4ca5-a9e4-0175332c1ead-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:09.734634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.734601    4909 generic.go:356] "Generic (PLEG): container finished" podID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerID="35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f" exitCode=0
Nov 03 12:53:09.734832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.734662    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"
Nov 03 12:53:09.734832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.734681    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" event={"ID":"7d902052-bb91-4ca5-a9e4-0175332c1ead","Type":"ContainerDied","Data":"35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f"}
Nov 03 12:53:09.734832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.734746    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m" event={"ID":"7d902052-bb91-4ca5-a9e4-0175332c1ead","Type":"ContainerDied","Data":"1106f7d267f6bd3c3b83ff5cff1bbd0f582e54bb9833734d0bfe943d4d2e6efa"}
Nov 03 12:53:09.734832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.734769    4909 scope.go:122] "RemoveContainer" containerID="35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f"
Nov 03 12:53:09.746529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.746509    4909 scope.go:122] "RemoveContainer" containerID="0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d"
Nov 03 12:53:09.755463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.755442    4909 scope.go:122] "RemoveContainer" containerID="35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f"
Nov 03 12:53:09.755790 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:53:09.755754    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f\": container with ID starting with 35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f not found: ID does not exist" containerID="35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f"
Nov 03 12:53:09.755895 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.755801    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f"} err="failed to get container status \"35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f\": rpc error: code = NotFound desc = could not find container \"35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f\": container with ID starting with 35292c3c74217e13256a62432efe0df0bd475a22b2d60e448a943971863dd42f not found: ID does not exist"
Nov 03 12:53:09.755895 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.755828    4909 scope.go:122] "RemoveContainer" containerID="0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d"
Nov 03 12:53:09.756083 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:53:09.756063    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d\": container with ID starting with 0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d not found: ID does not exist" containerID="0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d"
Nov 03 12:53:09.756129 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.756094    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d"} err="failed to get container status \"0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d\": rpc error: code = NotFound desc = could not find container \"0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d\": container with ID starting with 0210a0a2776bfcd9d2e9b02003bec641a63337474db31a105e5c67c12486d49d not found: ID does not exist"
Nov 03 12:53:09.886951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.886863    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"]
Nov 03 12:53:09.890543 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:09.890513    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-aa20ad-openshift-default-668fd78d49-sxm2m"]
Nov 03 12:53:10.432886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.432848    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"]
Nov 03 12:53:10.434111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434082    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy"
Nov 03 12:53:10.434111 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434109    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434137    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434144    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434168    4909 cpu_manager.go:414] "containerMap: removing container" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434176    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434204    4909 cpu_manager.go:414] "containerMap: removing container" podUID="1d0faaca-e103-423a-aa1c-d646d5600033" containerName="payload-processing"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434215    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0faaca-e103-423a-aa1c-d646d5600033" containerName="payload-processing"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434239    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f1c830fa-1dcf-48a4-adbe-927013af88bb" containerName="payload-processing"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434247    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c830fa-1dcf-48a4-adbe-927013af88bb" containerName="payload-processing"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434273    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434281    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434304    4909 cpu_manager.go:414] "containerMap: removing container" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy"
Nov 03 12:53:10.434307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434311    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434327    4909 cpu_manager.go:414] "containerMap: removing container" podUID="28854ff1-e49b-4aac-9b7a-386f471b01b0" containerName="payload-pre-processing"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434335    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28854ff1-e49b-4aac-9b7a-386f471b01b0" containerName="payload-pre-processing"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434366    4909 cpu_manager.go:414] "containerMap: removing container" podUID="62e94055-ab84-4679-b30f-4ce5c0ee93b8" containerName="payload-pre-processing"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434374    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e94055-ab84-4679-b30f-4ce5c0ee93b8" containerName="payload-pre-processing"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434652    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c830fa-1dcf-48a4-adbe-927013af88bb" containerName="payload-processing"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434677    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434686    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434718    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434727    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="28854ff1-e49b-4aac-9b7a-386f471b01b0" containerName="payload-pre-processing"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434738    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e94055-ab84-4679-b30f-4ce5c0ee93b8" containerName="payload-pre-processing"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434768    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01fbd75-25d5-4efd-8788-44a86c84cf1f" containerName="istio-proxy"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434778    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy"
Nov 03 12:53:10.434907 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.434804    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0faaca-e103-423a-aa1c-d646d5600033" containerName="payload-processing"
Nov 03 12:53:10.441247 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.441225    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.486769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.486744    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"]
Nov 03 12:53:10.489312 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.489291    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:53:10.489439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.489326    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:53:10.489439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.489375    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:53:10.489439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.489408    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1ab9c770bc22\""
Nov 03 12:53:10.489439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.489411    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:53:10.489439 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.489430    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6lkbc\""
Nov 03 12:53:10.527178 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.527153    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkgv\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-kube-api-access-rjkgv\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.527487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.527211    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-routing-overlay-volume\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.527487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.527291    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-provider-credentials\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.527487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.527403    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-plugins-config-volume\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.628681 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.628648    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-plugins-config-volume\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.628851 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.628756    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkgv\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-kube-api-access-rjkgv\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.628900 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.628855    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-routing-overlay-volume\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.628953 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.628899    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-provider-credentials\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.629457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.629431    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-routing-overlay-volume\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.629554 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.629441    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-plugins-config-volume\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.631300 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.631282    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-provider-credentials\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.681874 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.681843    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkgv\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-kube-api-access-rjkgv\") pod \"payload-processing-external-model-58c5dcc977-nc8tx\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") " pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:10.754889 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:10.754842    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:11.123986 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:11.123915    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" path="/var/lib/kubelet/pods/7d902052-bb91-4ca5-a9e4-0175332c1ead/volumes"
Nov 03 12:53:11.146331 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:11.146303    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"]
Nov 03 12:53:11.148366 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:53:11.148338    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f6867e_5b21_4653_a694_d4718be5492f.slice/crio-6838abd51decca6b027b016efd72e8a9312d3a9a4d0d77749706872d9fb86803 WatchSource:0}: Error finding container 6838abd51decca6b027b016efd72e8a9312d3a9a4d0d77749706872d9fb86803: Status 404 returned error can't find the container with id 6838abd51decca6b027b016efd72e8a9312d3a9a4d0d77749706872d9fb86803
Nov 03 12:53:11.798057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:11.798021    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx" event={"ID":"40f6867e-5b21-4653-a694-d4718be5492f","Type":"ContainerStarted","Data":"64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2"}
Nov 03 12:53:11.798450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:11.798084    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:11.798450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:11.798109    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx" event={"ID":"40f6867e-5b21-4653-a694-d4718be5492f","Type":"ContainerStarted","Data":"6838abd51decca6b027b016efd72e8a9312d3a9a4d0d77749706872d9fb86803"}
Nov 03 12:53:11.905343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:11.905296    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx" podStartSLOduration=1.905282386 podStartE2EDuration="1.905282386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:53:11.9025676 +0000 UTC m=+3740.915034263" watchObservedRunningTime="2026-11-03 12:53:11.905282386 +0000 UTC m=+3740.917749050"
Nov 03 12:53:22.806753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:22.806689    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:22.885550 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:22.885519    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"]
Nov 03 12:53:22.885761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:22.885723    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy" containerID="cri-o://13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d" gracePeriod=30
Nov 03 12:53:24.675826 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:24.675792    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:53:24.676229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:24.675858    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:53:28.148083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.148060    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:53:28.261079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261049    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-podinfo\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261144    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-kube-api-access-2c69g\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-kube-api-access-2c69g\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261221    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283fb1be-e666-43df-b3c2-8dcbc288ec06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283fb1be-e666-43df-b3c2-8dcbc288ec06-istiod-ca-cert\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261327 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261313    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-certs\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261388 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261372    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-data\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261408    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-socket\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261450    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-envoy\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261506 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261488    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-token\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-token\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261617 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261543    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-credential-socket\") pod \"283fb1be-e666-43df-b3c2-8dcbc288ec06\" (UID: \"283fb1be-e666-43df-b3c2-8dcbc288ec06\") "
Nov 03 12:53:28.261674 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261590    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-certs" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:28.261755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261689    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-socket" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:28.261755 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261741    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283fb1be-e666-43df-b3c2-8dcbc288ec06-istiod-ca-cert" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:28.261973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.261949    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-credential-socket" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:28.262075 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.262012    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-data" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:28.262156 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.262139    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.262220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.262158    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.262220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.262174    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.262220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.262187    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.262220 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.262200    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283fb1be-e666-43df-b3c2-8dcbc288ec06-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.263480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.263453    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-kube-api-access-2c69g" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "kube-api-access-2c69g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:28.263675 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.263657    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-podinfo" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:53:28.263780 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.263709    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-token" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:28.263878 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.263861    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-envoy" pod "283fb1be-e666-43df-b3c2-8dcbc288ec06" (UID: "283fb1be-e666-43df-b3c2-8dcbc288ec06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:28.355520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.355490    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:53:28.362869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.362847    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.362869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.362869    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c69g\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-kube-api-access-2c69g\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.363016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.362879    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.363016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.362888    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283fb1be-e666-43df-b3c2-8dcbc288ec06-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:28.907151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.907115    4909 generic.go:356] "Generic (PLEG): container finished" podID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerID="13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d" exitCode=0
Nov 03 12:53:28.907338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.907180    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"
Nov 03 12:53:28.907338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.907195    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" event={"ID":"283fb1be-e666-43df-b3c2-8dcbc288ec06","Type":"ContainerDied","Data":"13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d"}
Nov 03 12:53:28.907338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.907276    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj" event={"ID":"283fb1be-e666-43df-b3c2-8dcbc288ec06","Type":"ContainerDied","Data":"13e871a6368c3adc23d40668fc2168e2528b64f056012fda0f19ab0c1e39f220"}
Nov 03 12:53:28.907338 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.907301    4909 scope.go:122] "RemoveContainer" containerID="13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d"
Nov 03 12:53:28.918172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.918154    4909 scope.go:122] "RemoveContainer" containerID="b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda"
Nov 03 12:53:28.927641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.927618    4909 scope.go:122] "RemoveContainer" containerID="13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d"
Nov 03 12:53:28.928201 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:53:28.928174    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d\": container with ID starting with 13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d not found: ID does not exist" containerID="13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d"
Nov 03 12:53:28.928330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.928211    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d"} err="failed to get container status \"13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d\": rpc error: code = NotFound desc = could not find container \"13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d\": container with ID starting with 13b9d83a3f9796f62ba8977c14992461bd52ff0a1e370a2e38aaa409d87dd61d not found: ID does not exist"
Nov 03 12:53:28.928330 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.928236    4909 scope.go:122] "RemoveContainer" containerID="b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda"
Nov 03 12:53:28.928626 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:53:28.928607    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda\": container with ID starting with b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda not found: ID does not exist" containerID="b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda"
Nov 03 12:53:28.928732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:28.928637    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda"} err="failed to get container status \"b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda\": rpc error: code = NotFound desc = could not find container \"b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda\": container with ID starting with b62942f172e96bc642dd673d7173ef93792e673146f65fe65952f47d4c768eda not found: ID does not exist"
Nov 03 12:53:29.062761 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:29.062732    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"]
Nov 03 12:53:29.108585 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:29.108557    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c84ae1-openshift-default-5f46cffd57-tbpdj"]
Nov 03 12:53:29.123264 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:29.123238    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" path="/var/lib/kubelet/pods/283fb1be-e666-43df-b3c2-8dcbc288ec06/volumes"
Nov 03 12:53:36.146961 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.146927    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"]
Nov 03 12:53:36.147666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147645    4909 cpu_manager.go:414] "containerMap: removing container" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy"
Nov 03 12:53:36.147753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147666    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy"
Nov 03 12:53:36.147753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147680    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy"
Nov 03 12:53:36.147753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147686    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d902052-bb91-4ca5-a9e4-0175332c1ead" containerName="istio-proxy"
Nov 03 12:53:36.147753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147709    4909 cpu_manager.go:414] "containerMap: removing container" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy"
Nov 03 12:53:36.147753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147716    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy"
Nov 03 12:53:36.147950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147845    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy"
Nov 03 12:53:36.147950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.147857    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="679a8bb6-6ac5-455b-a303-0f6201ba1556" containerName="istio-proxy"
Nov 03 12:53:36.153035 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.153010    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.204471 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.204445    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"]
Nov 03 12:53:36.243203 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.243173    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-routing-overlay-volume\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.243363 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.243289    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.243408 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.243385    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8nj\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-kube-api-access-zz8nj\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.243532 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.243510    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-plugins-config-volume\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.246865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.246833    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6e306ea651ee\""
Nov 03 12:53:36.246865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.246848    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9ec01e4d84c9\""
Nov 03 12:53:36.344934 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.344904    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8nj\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-kube-api-access-zz8nj\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.345091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.345004    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-plugins-config-volume\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.345091 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.345026    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-routing-overlay-volume\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.345183 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.345127    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.345798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.345775    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-routing-overlay-volume\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.345908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.345799    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-plugins-config-volume\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.347745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.347726    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.396823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.396803    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8nj\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-kube-api-access-zz8nj\") pod \"payload-processing-external-model-599b55d695-nzcgg\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") " pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.466825 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.466797    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.833291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.833258    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"]
Nov 03 12:53:36.835363 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:53:36.835336    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3afccc9_9458_461e_ab3c_88780f5b1ce9.slice/crio-720b42d7fa9c2a5ca0076ddaf353c54f25027465c0173cd130612eb23ce7f167 WatchSource:0}: Error finding container 720b42d7fa9c2a5ca0076ddaf353c54f25027465c0173cd130612eb23ce7f167: Status 404 returned error can't find the container with id 720b42d7fa9c2a5ca0076ddaf353c54f25027465c0173cd130612eb23ce7f167
Nov 03 12:53:36.956406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.956369    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg" event={"ID":"e3afccc9-9458-461e-ab3c-88780f5b1ce9","Type":"ContainerStarted","Data":"e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83"}
Nov 03 12:53:36.956538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.956438    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:36.956538 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:36.956466    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg" event={"ID":"e3afccc9-9458-461e-ab3c-88780f5b1ce9","Type":"ContainerStarted","Data":"720b42d7fa9c2a5ca0076ddaf353c54f25027465c0173cd130612eb23ce7f167"}
Nov 03 12:53:37.064694 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:37.064645    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg" podStartSLOduration=1.06463086 podStartE2EDuration="1.06463086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:53:37.062205716 +0000 UTC m=+3766.074672378" watchObservedRunningTime="2026-11-03 12:53:37.06463086 +0000 UTC m=+3766.077097523"
Nov 03 12:53:39.564836 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:53:39.564801    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:53:47.965118 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:47.965088    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 12:53:48.119164 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:48.119132    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"]
Nov 03 12:53:53.119848 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.119787    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx" podUID="40f6867e-5b21-4653-a694-d4718be5492f" containerName="payload-processing" containerID="cri-o://64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2" gracePeriod=25
Nov 03 12:53:53.387307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.387242    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:53.535966 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.535937    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-routing-overlay-volume\") pod \"40f6867e-5b21-4653-a694-d4718be5492f\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") "
Nov 03 12:53:53.536138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.536008    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-provider-credentials\") pod \"40f6867e-5b21-4653-a694-d4718be5492f\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") "
Nov 03 12:53:53.536138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.536069    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-plugins-config-volume\") pod \"40f6867e-5b21-4653-a694-d4718be5492f\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") "
Nov 03 12:53:53.536138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.536111    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-kube-api-access-rjkgv\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-kube-api-access-rjkgv\") pod \"40f6867e-5b21-4653-a694-d4718be5492f\" (UID: \"40f6867e-5b21-4653-a694-d4718be5492f\") "
Nov 03 12:53:53.536370 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.536345    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-routing-overlay-volume" pod "40f6867e-5b21-4653-a694-d4718be5492f" (UID: "40f6867e-5b21-4653-a694-d4718be5492f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:53.536544 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.536517    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-plugins-config-volume" pod "40f6867e-5b21-4653-a694-d4718be5492f" (UID: "40f6867e-5b21-4653-a694-d4718be5492f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:53.536651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.536590    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:53.536651 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.536607    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f6867e-5b21-4653-a694-d4718be5492f-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:53.538229 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.538198    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-provider-credentials" pod "40f6867e-5b21-4653-a694-d4718be5492f" (UID: "40f6867e-5b21-4653-a694-d4718be5492f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:53.538318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.538286    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-kube-api-access-rjkgv" pod "40f6867e-5b21-4653-a694-d4718be5492f" (UID: "40f6867e-5b21-4653-a694-d4718be5492f"). InnerVolumeSpecName "kube-api-access-rjkgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:53.638138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.638063    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:53.638138 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:53.638091    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkgv\" (UniqueName: \"kubernetes.io/projected/40f6867e-5b21-4653-a694-d4718be5492f-kube-api-access-rjkgv\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:54.069962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.069931    4909 generic.go:356] "Generic (PLEG): container finished" podID="40f6867e-5b21-4653-a694-d4718be5492f" containerID="64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2" exitCode=0
Nov 03 12:53:54.070275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.069983    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"
Nov 03 12:53:54.070275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.070018    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx" event={"ID":"40f6867e-5b21-4653-a694-d4718be5492f","Type":"ContainerDied","Data":"64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2"}
Nov 03 12:53:54.070275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.070071    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx" event={"ID":"40f6867e-5b21-4653-a694-d4718be5492f","Type":"ContainerDied","Data":"6838abd51decca6b027b016efd72e8a9312d3a9a4d0d77749706872d9fb86803"}
Nov 03 12:53:54.070275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.070095    4909 scope.go:122] "RemoveContainer" containerID="64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2"
Nov 03 12:53:54.079823 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.079806    4909 scope.go:122] "RemoveContainer" containerID="64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2"
Nov 03 12:53:54.080117 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:53:54.080094    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2\": container with ID starting with 64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2 not found: ID does not exist" containerID="64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2"
Nov 03 12:53:54.080188 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.080136    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2"} err="failed to get container status \"64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2\": rpc error: code = NotFound desc = could not find container \"64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2\": container with ID starting with 64d8b36049d77d263230079f2e3a50030998b4fe461338c01649a917c00dd7c2 not found: ID does not exist"
Nov 03 12:53:54.220630 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.220604    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"]
Nov 03 12:53:54.266303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:54.266277    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58c5dcc977-nc8tx"]
Nov 03 12:53:55.107256 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.107223    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"]
Nov 03 12:53:55.107861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.107844    4909 cpu_manager.go:414] "containerMap: removing container" podUID="40f6867e-5b21-4653-a694-d4718be5492f" containerName="payload-processing"
Nov 03 12:53:55.107913 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.107863    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f6867e-5b21-4653-a694-d4718be5492f" containerName="payload-processing"
Nov 03 12:53:55.107979 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.107969    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f6867e-5b21-4653-a694-d4718be5492f" containerName="payload-processing"
Nov 03 12:53:55.108018 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.107987    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="283fb1be-e666-43df-b3c2-8dcbc288ec06" containerName="istio-proxy"
Nov 03 12:53:55.112806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.112788    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.125362 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.125341    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f6867e-5b21-4653-a694-d4718be5492f" path="/var/lib/kubelet/pods/40f6867e-5b21-4653-a694-d4718be5492f/volumes"
Nov 03 12:53:55.161649 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.161624    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg57t\""
Nov 03 12:53:55.211918 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.211892    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"]
Nov 03 12:53:55.256267 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.256238    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.256690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.256352    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.256690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.256403    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.256690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.256647    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.256962 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.256924    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.257110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.256963    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.257110 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.257065    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.257218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.257122    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9p2\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-kube-api-access-cf9p2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.257218 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.257184    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.358572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358488    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9p2\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-kube-api-access-cf9p2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.358572 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358531    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.358855 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358651    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.358855 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358726    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.358855 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358777    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358857    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358914    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358918    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358939    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359026 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.358992    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.359153    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359295 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.359190    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359418 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.359354    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.359480 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.359467    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.361098 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.361074    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.361463 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.361447    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.412237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.412214    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9p2\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-kube-api-access-cf9p2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.412536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.412514    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dmgvs\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.427879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.427857    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:55.844095 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.844071    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"]
Nov 03 12:53:55.845092 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:53:55.845057    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf516db_3ff5_4e54_b0f4_2bf041f6b161.slice/crio-3b1d679b5e1caf2b1c1091422f11ede8413beaf0b249eea5af3fc8b76e60ccd6 WatchSource:0}: Error finding container 3b1d679b5e1caf2b1c1091422f11ede8413beaf0b249eea5af3fc8b76e60ccd6: Status 404 returned error can't find the container with id 3b1d679b5e1caf2b1c1091422f11ede8413beaf0b249eea5af3fc8b76e60ccd6
Nov 03 12:53:55.847832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.847802    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:53:55.847910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.847869    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:53:55.847910 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:55.847899    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:53:56.087310 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:56.087268    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" event={"ID":"2bf516db-3ff5-4e54-b0f4-2bf041f6b161","Type":"ContainerStarted","Data":"d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd"}
Nov 03 12:53:56.087508 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:56.087372    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" event={"ID":"2bf516db-3ff5-4e54-b0f4-2bf041f6b161","Type":"ContainerStarted","Data":"3b1d679b5e1caf2b1c1091422f11ede8413beaf0b249eea5af3fc8b76e60ccd6"}
Nov 03 12:53:56.196121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:56.196077    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podStartSLOduration=1.196062779 podStartE2EDuration="1.196062779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:53:56.19362169 +0000 UTC m=+3785.206088353" watchObservedRunningTime="2026-11-03 12:53:56.196062779 +0000 UTC m=+3785.208529442"
Nov 03 12:53:56.429847 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:56.429480    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:53:56.431987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:56.431948    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:53:56.432180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:56.432021    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:53:57.428486 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:57.428449    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:53:57.428666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:57.428504    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:53:58.429152 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:58.429121    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:53:58.429534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:58.429175    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:53:59.428450 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:59.428412    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:53:59.428633 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:53:59.428473    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:54:00.429109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:00.429077    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:54:00.429495 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:00.429136    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:54:01.428641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:01.428605    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:54:01.428833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:01.428664    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:54:02.428782 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:02.428748    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:54:02.429169 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:02.428798    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:54:03.430666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:03.430625    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dmgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 12:54:03.431216 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:03.430685    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 12:54:04.478753 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:04.478718    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:54:04.479208 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:04.479192    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:54:04.480126 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:04.480097    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 12:54:08.280606 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.280572    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"]
Nov 03 12:54:08.285115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.285094    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338399    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-data\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338566 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338470    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-credential-socket\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338643 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338581    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbxt\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-kube-api-access-4vbxt\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338638    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-envoy\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338735    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istiod-ca-cert\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338867 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338795    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-token\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338925 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338876    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-podinfo\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338924    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-socket\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.338984 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.338949    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-certs\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.366690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.366665    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b737fe-openshift-default-dockercfg-scrht\""
Nov 03 12:54:08.383677 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.383649    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"]
Nov 03 12:54:08.453798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.453762    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-token\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.456415 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456369    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-podinfo\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.456579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456461    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-socket\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.456579 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456486    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-certs\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.456750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456601    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-data\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.456750 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456647    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-credential-socket\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.457043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456728    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbxt\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-kube-api-access-4vbxt\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.457043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456802    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-envoy\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.457043 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.456975    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istiod-ca-cert\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.466893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.457813    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istiod-ca-cert\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.466893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.458500    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-data\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.466893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.458770    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-socket\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.466893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.459008    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-certs\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.466893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.459554    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-credential-socket\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.466893 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.462076    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-podinfo\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.499073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.499036    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-envoy\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.511087 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.511057    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-token\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.512534 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.512506    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbxt\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-kube-api-access-4vbxt\") pod \"e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") " pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:08.598825 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:08.598750    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:09.053589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.053558    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"]
Nov 03 12:54:09.054881 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:54:09.054855    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93edf994_9d3f_4af0_a998_6c8a2cb9a2c2.slice/crio-2b0a951e72ad8304c6f5c34e26c504b26079fc58fd49bcce399dbbab556554cf WatchSource:0}: Error finding container 2b0a951e72ad8304c6f5c34e26c504b26079fc58fd49bcce399dbbab556554cf: Status 404 returned error can't find the container with id 2b0a951e72ad8304c6f5c34e26c504b26079fc58fd49bcce399dbbab556554cf
Nov 03 12:54:09.057417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.057382    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:09.057524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.057459    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:09.057524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.057506    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:09.171356 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.171299    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" event={"ID":"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2","Type":"ContainerStarted","Data":"82842d7e672a6524c760ca88d152b7453aabc92e8170c64a603f84ce29420a06"}
Nov 03 12:54:09.171524 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.171381    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" event={"ID":"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2","Type":"ContainerStarted","Data":"2b0a951e72ad8304c6f5c34e26c504b26079fc58fd49bcce399dbbab556554cf"}
Nov 03 12:54:09.279920 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.279862    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podStartSLOduration=1.279843262 podStartE2EDuration="1.279843262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:09.278023094 +0000 UTC m=+3798.290489759" watchObservedRunningTime="2026-11-03 12:54:09.279843262 +0000 UTC m=+3798.292309926"
Nov 03 12:54:09.565652 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:09.565618    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:54:09.599319 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:09.599282    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:10.605050 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:10.605015    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:11.186879 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:11.186849    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:11.187863 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:11.187843    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:54:11.526003 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:11.525970    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"]
Nov 03 12:54:15.659120 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.659087    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"]
Nov 03 12:54:15.664451 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.664425    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:15.711448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.711416    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"]
Nov 03 12:54:15.740455 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.740424    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570ef14d-3b08-46d6-840d-7281aa3b6e21-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:15.740588 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.740485    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lsv4\" (UniqueName: \"kubernetes.io/projected/570ef14d-3b08-46d6-840d-7281aa3b6e21-kube-api-access-7lsv4\") pod \"payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:15.753845 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.753811    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b737fe\""
Nov 03 12:54:15.753845 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.753829    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b737fe-dockercfg-tsjg7\""
Nov 03 12:54:15.841417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.841389    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570ef14d-3b08-46d6-840d-7281aa3b6e21-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:15.841536 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.841427    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lsv4\" (UniqueName: \"kubernetes.io/projected/570ef14d-3b08-46d6-840d-7281aa3b6e21-kube-api-access-7lsv4\") pod \"payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:15.842083 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.842062    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570ef14d-3b08-46d6-840d-7281aa3b6e21-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:15.893983 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.893961    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lsv4\" (UniqueName: \"kubernetes.io/projected/570ef14d-3b08-46d6-840d-7281aa3b6e21-kube-api-access-7lsv4\") pod \"payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:15.979606 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:15.979559    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:16.345492 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:16.345466    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"]
Nov 03 12:54:16.346936 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:54:16.346914    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570ef14d_3b08_46d6_840d_7281aa3b6e21.slice/crio-af9374e6d2c524eb4df8edc686f4b0df70226e1108c7f5ffa886c56b92c4f602 WatchSource:0}: Error finding container af9374e6d2c524eb4df8edc686f4b0df70226e1108c7f5ffa886c56b92c4f602: Status 404 returned error can't find the container with id af9374e6d2c524eb4df8edc686f4b0df70226e1108c7f5ffa886c56b92c4f602
Nov 03 12:54:17.253275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:17.253232    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc" event={"ID":"570ef14d-3b08-46d6-840d-7281aa3b6e21","Type":"ContainerStarted","Data":"aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b"}
Nov 03 12:54:17.253789 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:17.253298    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc" event={"ID":"570ef14d-3b08-46d6-840d-7281aa3b6e21","Type":"ContainerStarted","Data":"af9374e6d2c524eb4df8edc686f4b0df70226e1108c7f5ffa886c56b92c4f602"}
Nov 03 12:54:17.253789 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:17.253520    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:17.360318 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:17.360264    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc" podStartSLOduration=2.360249643 podStartE2EDuration="2.360249643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:17.356999824 +0000 UTC m=+3806.369466488" watchObservedRunningTime="2026-11-03 12:54:17.360249643 +0000 UTC m=+3806.372716308"
Nov 03 12:54:22.594809 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:22.594775    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:54:26.548259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.548207    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw" podUID="79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" containerName="main" containerID="cri-o://1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9" gracePeriod=45
Nov 03 12:54:26.827398 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.827373    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:54:26.972147 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972115    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tls-certs\") pod \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") "
Nov 03 12:54:26.972329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972208    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-model-cache\") pod \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") "
Nov 03 12:54:26.972329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972256    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-dshm\") pod \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") "
Nov 03 12:54:26.972329 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972290    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-kube-api-access-t5994\" (UniqueName: \"kubernetes.io/projected/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-kube-api-access-t5994\") pod \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") "
Nov 03 12:54:26.972438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972385    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tmp-dir\") pod \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") "
Nov 03 12:54:26.972438 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972401    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-home\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-home\") pod \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\" (UID: \"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6\") "
Nov 03 12:54:26.972537 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972461    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-model-cache" pod "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" (UID: "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:54:26.972721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972673    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tmp-dir" pod "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" (UID: "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:54:26.972800 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972756    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-home" pod "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" (UID: "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:54:26.972988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972970    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:26.973088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.972993    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:26.973088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.973005    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:26.974528 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.974502    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-dshm" pod "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" (UID: "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:54:26.974638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.974555    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tls-certs" pod "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" (UID: "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:54:26.974638 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:26.974591    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-kube-api-access-t5994" pod "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" (UID: "79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6"). InnerVolumeSpecName "kube-api-access-t5994". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:54:27.074273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.074202    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5994\" (UniqueName: \"kubernetes.io/projected/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-kube-api-access-t5994\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:27.074273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.074230    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:27.074273 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.074240    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:27.335529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.335448    4909 generic.go:356] "Generic (PLEG): container finished" podID="79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" containerID="1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9" exitCode=0
Nov 03 12:54:27.335529 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.335492    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"
Nov 03 12:54:27.335737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.335524    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw" event={"ID":"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6","Type":"ContainerDied","Data":"1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9"}
Nov 03 12:54:27.335737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.335574    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw" event={"ID":"79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6","Type":"ContainerDied","Data":"109fdb8702371231333c621b14159aa3ef989c1d4094e5bfdea6688a3fe6e4e8"}
Nov 03 12:54:27.335737 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.335593    4909 scope.go:122] "RemoveContainer" containerID="1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9"
Nov 03 12:54:27.345140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.345120    4909 scope.go:122] "RemoveContainer" containerID="1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9"
Nov 03 12:54:27.345399 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:27.345377    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9\": container with ID starting with 1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9 not found: ID does not exist" containerID="1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9"
Nov 03 12:54:27.345444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.345409    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9"} err="failed to get container status \"1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9\": rpc error: code = NotFound desc = could not find container \"1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9\": container with ID starting with 1472d0219d03185c5c45c9adbec815b878eb7eb0d738f22b56b41b8e2c4af2c9 not found: ID does not exist"
Nov 03 12:54:27.486833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.486805    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"]
Nov 03 12:54:27.489490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:27.489461    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/auth-test-model-6fb3a8-kserve-56f84c86fc-wrntw"]
Nov 03 12:54:28.263403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:28.263371    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 12:54:29.124016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:29.123983    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" path="/var/lib/kubelet/pods/79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6/volumes"
Nov 03 12:54:39.564391 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:39.564315    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:54:41.241297 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.241263    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"]
Nov 03 12:54:41.241972 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.241953    4909 cpu_manager.go:414] "containerMap: removing container" podUID="79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" containerName="main"
Nov 03 12:54:41.242045 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.241973    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" containerName="main"
Nov 03 12:54:41.242121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.242112    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="79de6af0-d2d4-43d2-97e7-d8c8d70bf6d6" containerName="main"
Nov 03 12:54:41.247215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.247192    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.298274 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.298251    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"]
Nov 03 12:54:41.329269 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.329238    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7g25\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-kube-api-access-h7g25\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.329381 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.329312    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-plugins-config-volume\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.329445 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.329434    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-routing-overlay-volume\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.329489 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.329467    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-provider-credentials\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.430833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.430794    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7g25\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-kube-api-access-h7g25\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.430999 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.430894    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-plugins-config-volume\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.431071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.431051    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-routing-overlay-volume\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.431142 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.431116    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-provider-credentials\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.431763 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.431742    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-plugins-config-volume\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.431934 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.431764    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-routing-overlay-volume\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.433719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.433689    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-provider-credentials\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.483028 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.483004    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7g25\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-kube-api-access-h7g25\") pod \"payload-processing-external-model-57cf66cd74-fj4g9\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") " pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.561634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.561574    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:41.938943 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:41.938920    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"]
Nov 03 12:54:41.941900 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:54:41.941871    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3e8457_0670_4daa_8354_e75592d5b17a.slice/crio-d182e5f9aa8f968bd9eb99c6323dafbc8501b0784f1f6f549ed34e4b58269829 WatchSource:0}: Error finding container d182e5f9aa8f968bd9eb99c6323dafbc8501b0784f1f6f549ed34e4b58269829: Status 404 returned error can't find the container with id d182e5f9aa8f968bd9eb99c6323dafbc8501b0784f1f6f549ed34e4b58269829
Nov 03 12:54:42.424852 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:42.424758    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" event={"ID":"cc3e8457-0670-4daa-8354-e75592d5b17a","Type":"ContainerStarted","Data":"5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170"}
Nov 03 12:54:42.425240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:42.424854    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:54:42.425240 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:42.424880    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" event={"ID":"cc3e8457-0670-4daa-8354-e75592d5b17a","Type":"ContainerStarted","Data":"d182e5f9aa8f968bd9eb99c6323dafbc8501b0784f1f6f549ed34e4b58269829"}
Nov 03 12:54:42.537361 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:42.537310    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podStartSLOduration=1.5372965280000002 podStartE2EDuration="1.537296528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:42.535035635 +0000 UTC m=+3831.547502297" watchObservedRunningTime="2026-11-03 12:54:42.537296528 +0000 UTC m=+3831.549763192"
Nov 03 12:54:51.149172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.149143    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"]
Nov 03 12:54:51.154678 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.154658    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.221357 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.221307    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"]
Nov 03 12:54:51.294927 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.294885    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d126a6\""
Nov 03 12:54:51.295306 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.295274    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d126a6-dockercfg-4qr6n\""
Nov 03 12:54:51.313121 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.313088    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e41f9b2-8122-4835-9bcd-edd411486a7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.313275 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.313258    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s75q\" (UniqueName: \"kubernetes.io/projected/3e41f9b2-8122-4835-9bcd-edd411486a7a-kube-api-access-2s75q\") pod \"payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.414270 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.414186    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s75q\" (UniqueName: \"kubernetes.io/projected/3e41f9b2-8122-4835-9bcd-edd411486a7a-kube-api-access-2s75q\") pod \"payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.414448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.414354    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e41f9b2-8122-4835-9bcd-edd411486a7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.415115 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.415088    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e41f9b2-8122-4835-9bcd-edd411486a7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.466798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.466763    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s75q\" (UniqueName: \"kubernetes.io/projected/3e41f9b2-8122-4835-9bcd-edd411486a7a-kube-api-access-2s75q\") pod \"payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.476504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.476480    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:51.844484 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:51.844449    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"]
Nov 03 12:54:51.845658 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:54:51.845630    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e41f9b2_8122_4835_9bcd_edd411486a7a.slice/crio-e9ad48f8e76d00f6f1f321b844ecb2a4a4e17f18e7087ad06c0df047c99d183d WatchSource:0}: Error finding container e9ad48f8e76d00f6f1f321b844ecb2a4a4e17f18e7087ad06c0df047c99d183d: Status 404 returned error can't find the container with id e9ad48f8e76d00f6f1f321b844ecb2a4a4e17f18e7087ad06c0df047c99d183d
Nov 03 12:54:52.493249 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:52.493215    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6" event={"ID":"3e41f9b2-8122-4835-9bcd-edd411486a7a","Type":"ContainerStarted","Data":"c801cfd4b132aa95bdf51b4bd79c32ce5fd2b3173fd4a9557ca3aa4d4f4e7a65"}
Nov 03 12:54:52.493634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:52.493265    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6" event={"ID":"3e41f9b2-8122-4835-9bcd-edd411486a7a","Type":"ContainerStarted","Data":"e9ad48f8e76d00f6f1f321b844ecb2a4a4e17f18e7087ad06c0df047c99d183d"}
Nov 03 12:54:52.493634 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:52.493319    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:54:52.598639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:52.598591    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6" podStartSLOduration=1.5985772539999998 podStartE2EDuration="1.598577254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:52.597321631 +0000 UTC m=+3841.609788293" watchObservedRunningTime="2026-11-03 12:54:52.598577254 +0000 UTC m=+3841.611043919"
Nov 03 12:54:53.434237 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:53.434201    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:54:55.543906 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.543875    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"]
Nov 03 12:54:55.549316 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.549295    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.590228 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.590200    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"]
Nov 03 12:54:55.594332 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.594314    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.646123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.646094    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"]
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664641    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hg6\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-kube-api-access-x2hg6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664690    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664726    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664864    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664905    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664938    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664967    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.664995    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665136    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665291 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665238    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665357    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665413    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665477    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665513    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qzr\" (UniqueName: \"kubernetes.io/projected/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-kube-api-access-g9qzr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665540    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665620    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665636    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33323943-1d1c-4ad0-aa72-c73b1e72758a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.665838 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.665675    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.692806 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.692778    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"]
Nov 03 12:54:55.766384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.766350    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.766646 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.766625    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.766811 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.766792    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qzr\" (UniqueName: \"kubernetes.io/projected/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-kube-api-access-g9qzr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.766959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.766935    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.767088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.766995    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767024    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33323943-1d1c-4ad0-aa72-c73b1e72758a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767088 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767079    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767117    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hg6\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-kube-api-access-x2hg6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767132    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767156    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.767266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767184    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767266 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767240    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767268    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767297    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767334    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767363    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767388    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767393    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767448    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767517    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.767989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.767569    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.767989 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.766833    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.768104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.768052    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.768104 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.768090    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.768320 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.768295    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.768442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.768332    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.768513 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.768488    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33323943-1d1c-4ad0-aa72-c73b1e72758a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.769048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.769022    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.771044 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.770995    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.771230 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.771206    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.771313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.771215    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.771887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.771866    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.819656 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.819606    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.819769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.819745    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qzr\" (UniqueName: \"kubernetes.io/projected/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-kube-api-access-g9qzr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.820114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.820096    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e0502c-2cd5-4ee7-a0cf-6807f9c3f597-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v5l4l\" (UID: \"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.820746 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.820729    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hg6\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-kube-api-access-x2hg6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zjqz9\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:55.863406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.863376    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:55.907117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:55.907087    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:56.278285 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.278251    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"]
Nov 03 12:54:56.315323 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:54:56.315292    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e0502c_2cd5_4ee7_a0cf_6807f9c3f597.slice/crio-191797de3ba05c915658fd6d99b8b17a9991b2e7dda3857d6454288a4778e505 WatchSource:0}: Error finding container 191797de3ba05c915658fd6d99b8b17a9991b2e7dda3857d6454288a4778e505: Status 404 returned error can't find the container with id 191797de3ba05c915658fd6d99b8b17a9991b2e7dda3857d6454288a4778e505
Nov 03 12:54:56.317875 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.317847    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:56.317941 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.317911    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:56.317988 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.317938    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:56.326599 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.326577    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"]
Nov 03 12:54:56.329162 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:54:56.329138    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33323943_1d1c_4ad0_aa72_c73b1e72758a.slice/crio-e53463e0201ec630cd125ca7af48d85ebf6e13d99d829e37cac8fbba00c0c587 WatchSource:0}: Error finding container e53463e0201ec630cd125ca7af48d85ebf6e13d99d829e37cac8fbba00c0c587: Status 404 returned error can't find the container with id e53463e0201ec630cd125ca7af48d85ebf6e13d99d829e37cac8fbba00c0c587
Nov 03 12:54:56.331664 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.331630    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:56.331769 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.331729    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:56.331824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.331774    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:54:56.520399 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.520366    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" event={"ID":"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597","Type":"ContainerStarted","Data":"3da5d832a9a6346c2e9c2a25249b17f6d8ac638eab4b9bf9e81da37a56d86626"}
Nov 03 12:54:56.520589 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.520425    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" event={"ID":"85e0502c-2cd5-4ee7-a0cf-6807f9c3f597","Type":"ContainerStarted","Data":"191797de3ba05c915658fd6d99b8b17a9991b2e7dda3857d6454288a4778e505"}
Nov 03 12:54:56.522109 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.522085    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" event={"ID":"33323943-1d1c-4ad0-aa72-c73b1e72758a","Type":"ContainerStarted","Data":"cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68"}
Nov 03 12:54:56.522213 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.522128    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" event={"ID":"33323943-1d1c-4ad0-aa72-c73b1e72758a","Type":"ContainerStarted","Data":"e53463e0201ec630cd125ca7af48d85ebf6e13d99d829e37cac8fbba00c0c587"}
Nov 03 12:54:56.629768 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.629642    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podStartSLOduration=1.629628312 podStartE2EDuration="1.629628312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:56.627634107 +0000 UTC m=+3845.640100769" watchObservedRunningTime="2026-11-03 12:54:56.629628312 +0000 UTC m=+3845.642095042"
Nov 03 12:54:56.738956 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.738890    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podStartSLOduration=1.738869019 podStartE2EDuration="1.738869019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:56.734841633 +0000 UTC m=+3845.747308327" watchObservedRunningTime="2026-11-03 12:54:56.738869019 +0000 UTC m=+3845.751335683"
Nov 03 12:54:56.864470 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.864441    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:54:56.907828 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:56.907756    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:54:57.886944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:57.886890    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 12:54:57.887466 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:57.886983    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 12:54:57.909641 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:57.909602    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 12:54:57.909813 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:57.909663    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 12:54:58.205014 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:58.204974    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:58.205014 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:58.205016    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:58.205299 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:58.205097    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:54:58.705079287 +0000 UTC m=+3847.717545935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:58.711730 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:58.710459    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:58.711730 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:58.710493    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:58.711730 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:58.710550    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:54:59.710536097 +0000 UTC m=+3848.723002743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:58.866721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:58.866133    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 12:54:58.866721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:58.866220    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 12:54:58.908539 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:58.908506    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 12:54:58.909135 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:58.909104    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 12:54:59.736849 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:59.736741    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:59.736849 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:59.736780    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:59.736849 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:54:59.736848    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:55:01.736829546 +0000 UTC m=+3850.749296206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:54:59.866117 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:59.866080    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 12:54:59.866417 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:59.866380    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 12:54:59.908085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:59.908052    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 12:54:59.908322 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:54:59.908303    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 12:55:00.866136 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:00.866085    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 12:55:00.866776 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:00.866744    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 12:55:00.907815 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:00.907786    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 12:55:00.908057 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:00.908032    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 12:55:01.829428 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:01.829385    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:01.829428 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:01.829430    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:01.829692 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:01.829487    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:55:05.829469336 +0000 UTC m=+3854.841935977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:01.864114 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:01.864075    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 12:55:01.864387 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:01.864355    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 12:55:01.909720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:01.909097    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 12:55:01.909720 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:01.909184    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 12:55:02.864190 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:02.864153    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 12:55:02.864395 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:02.864216    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 12:55:02.909321 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:02.909290    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 12:55:02.909482 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:02.909331    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 12:55:03.432454 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:03.432413    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:55:03.502886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:03.502850    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 12:55:03.864011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:03.863970    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 12:55:03.864225 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:03.864047    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 12:55:03.907886 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:03.907843    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 12:55:03.908063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:03.907901    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 12:55:05.865250 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:05.865205    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:05.865845 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:05.865296    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:05.908855 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:05.908824    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:05.909016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:05.908871    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:05.909512 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:05.909479    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:05.909512 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:05.909509    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:05.909689 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:05.909556    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:55:13.9095427 +0000 UTC m=+3862.922009344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:06.906887 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:06.906842    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:06.907313 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:06.906897    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:06.949908 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:06.949835    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:06.950096 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:06.949933    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:07.936494 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.936435    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:07.937024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.936521    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:07.941639 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.941603    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:55:07.942033 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.942012    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:55:07.942822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.942801    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l"
Nov 03 12:55:07.946766 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.946747    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:55:07.947016 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.946995    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:55:07.948496 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:07.948471    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 12:55:09.564505 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:09.564470    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:55:13.432252 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:13.432214    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:55:13.929852 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:13.929821    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:13.929852 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:13.929852    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:13.930065 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:13.929908    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:55:29.929892028 +0000 UTC m=+3878.942358670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:23.432807 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:23.432772    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:55:29.951463 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:29.951432    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:29.951463 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:29.951466    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:29.952026 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:29.951531    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:56:01.95151618 +0000 UTC m=+3910.963982833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:55:33.432526 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:33.432489    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:55:35.772557 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.772523    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"]
Nov 03 12:55:35.777166 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.777146    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.826189 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.826164    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"payload-processing-external-model-plugins-e2e-extmod-b-94e247\""
Nov 03 12:55:35.826383 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.826358    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"openshift-service-ca.crt\""
Nov 03 12:55:35.826383 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.826375    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"kube-root-ca.crt\""
Nov 03 12:55:35.828020 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.827999    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.828124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.828052    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.828124 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.828098    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7799\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-kube-api-access-h7799\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.828303 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.828282    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.845343 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.845321    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"agc-cred-adddf2a8da5e\""
Nov 03 12:55:35.845456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.845349    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"agc-cred-8bea6cf1ca2b\""
Nov 03 12:55:35.845456 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.845362    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"routing-overlay\""
Nov 03 12:55:35.872179 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.872159    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"agc-cred-fdabd14ba96e\""
Nov 03 12:55:35.872406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.872387    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"payload-processing-external-model-e2e--332b9201-dockercfg-9cmzl\""
Nov 03 12:55:35.875428 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.875407    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"]
Nov 03 12:55:35.929612 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.929586    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.929757 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.929741    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.929817 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.929791    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.929862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.929816    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7799\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-kube-api-access-h7799\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.930448 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.930428    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.930545 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.930452    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.932393 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.932374    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:35.982073 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:35.982048    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7799\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-kube-api-access-h7799\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:36.091315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:36.091258    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:36.461092 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:36.461067    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"]
Nov 03 12:55:36.462775 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:55:36.462736    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5a6b84_7986_4f26_8708_b4922cf636c8.slice/crio-95a2a464781f1ee528155443af8c9cb72f071634e5cf50e4bf591023aab465a1 WatchSource:0}: Error finding container 95a2a464781f1ee528155443af8c9cb72f071634e5cf50e4bf591023aab465a1: Status 404 returned error can't find the container with id 95a2a464781f1ee528155443af8c9cb72f071634e5cf50e4bf591023aab465a1
Nov 03 12:55:36.892241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:36.892161    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg" event={"ID":"3f5a6b84-7986-4f26-8708-b4922cf636c8","Type":"ContainerStarted","Data":"463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054"}
Nov 03 12:55:36.892241 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:36.892214    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg" event={"ID":"3f5a6b84-7986-4f26-8708-b4922cf636c8","Type":"ContainerStarted","Data":"95a2a464781f1ee528155443af8c9cb72f071634e5cf50e4bf591023aab465a1"}
Nov 03 12:55:36.892654 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:36.892359    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:36.997668 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:36.997615    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg" podStartSLOduration=1.997603115 podStartE2EDuration="1.997603115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:55:36.996725395 +0000 UTC m=+3886.009192053" watchObservedRunningTime="2026-11-03 12:55:36.997603115 +0000 UTC m=+3886.010069777"
Nov 03 12:55:39.563982 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:55:39.563946    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:55:43.432523 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:43.432439    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:55:44.053952 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:44.053915    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:55:47.901658 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:47.901622    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 12:55:51.630328 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:51.630299    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:55:51.638151 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:51.638125    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 12:55:53.432062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:55:53.432027    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:01.964659 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:56:01.964583    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:56:01.964659 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:56:01.964616    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:56:01.965173 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:56:01.964685    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:57:05.964666633 +0000 UTC m=+3974.977133274 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:56:03.431726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:03.431666    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:09.119490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:09.119451    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:09.564535 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:56:09.564496    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:56:19.119609 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:19.119567    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:19.532127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:19.532097    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:56:29.119824 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:29.119778    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:39.120959 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:39.120919    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:39.564450 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:56:39.564412    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:56:44.671577 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:44.671544    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"]
Nov 03 12:56:44.673600 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:44.673561    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:46.251421 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.251384    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"]
Nov 03 12:56:46.255315 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.255298    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.353293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.353255    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"]
Nov 03 12:56:46.385862 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.385822    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-532442-openshift-default-dockercfg-pw889\""
Nov 03 12:56:46.466082 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.466044    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f9e32-8924-4f74-8623-00be96ff490b-istio-podinfo\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.466373 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.466342    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-socket\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.466549 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.466521    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f9e32-8924-4f74-8623-00be96ff490b-istiod-ca-cert\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.466797 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.466769    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgngz\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-kube-api-access-xgngz\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.467009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.466994    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-certs\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.467170 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.467148    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-credential-socket\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.467442 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.467419    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-envoy\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.467692 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.467671    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-data\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.467990 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.467972    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-istio-token\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569313    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-istio-token\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569364    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f9e32-8924-4f74-8623-00be96ff490b-istio-podinfo\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569394 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569385    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-socket\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569626 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569538    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f9e32-8924-4f74-8623-00be96ff490b-istiod-ca-cert\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569669 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569647    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgngz\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-kube-api-access-xgngz\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569778 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569758    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-certs\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569791    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-socket\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569812    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-credential-socket\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.569944 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569917    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-envoy\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.570005 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.569946    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-data\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.570101 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.570078    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-certs\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.570293 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.570263    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f9e32-8924-4f74-8623-00be96ff490b-istiod-ca-cert\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.570385 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.570263    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-credential-socket\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.570449 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.570377    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-data\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.572066 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.572044    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-envoy\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.572334 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.572316    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f9e32-8924-4f74-8623-00be96ff490b-istio-podinfo\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.621715 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.621682    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgngz\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-kube-api-access-xgngz\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.622008 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.621985    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-istio-token\") pod \"e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") " pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:46.871358 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:46.871269    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:47.332163 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.332133    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"]
Nov 03 12:56:47.333395 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:56:47.333368    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73f9e32_8924_4f74_8623_00be96ff490b.slice/crio-840ad01132615514d2c058e0630360198146f96ff4c5df872eb46cab44693435 WatchSource:0}: Error finding container 840ad01132615514d2c058e0630360198146f96ff4c5df872eb46cab44693435: Status 404 returned error can't find the container with id 840ad01132615514d2c058e0630360198146f96ff4c5df872eb46cab44693435
Nov 03 12:56:47.335973 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.335938    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:56:47.336064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.336007    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:56:47.336064 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.336042    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:56:47.506403 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.506360    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" event={"ID":"b73f9e32-8924-4f74-8623-00be96ff490b","Type":"ContainerStarted","Data":"8d8e9a4dbf3861a82d056dc853655b380f67017aede09dab51e75dd9de84161a"}
Nov 03 12:56:47.506582 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.506431    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" event={"ID":"b73f9e32-8924-4f74-8623-00be96ff490b","Type":"ContainerStarted","Data":"840ad01132615514d2c058e0630360198146f96ff4c5df872eb46cab44693435"}
Nov 03 12:56:47.617636 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.617533    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podStartSLOduration=1.617517113 podStartE2EDuration="1.617517113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:56:47.612862859 +0000 UTC m=+3956.625329535" watchObservedRunningTime="2026-11-03 12:56:47.617517113 +0000 UTC m=+3956.629983775"
Nov 03 12:56:47.871819 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:47.871748    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:48.877140 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:48.877112    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:49.519987 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:49.519960    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:49.520678 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:49.520657    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:56:49.675832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:49.675763    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing" containerID="cri-o://5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170" gracePeriod=25
Nov 03 12:56:50.000102 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.000076    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:56:50.164063 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.163964    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-provider-credentials\") pod \"cc3e8457-0670-4daa-8354-e75592d5b17a\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") "
Nov 03 12:56:50.164301 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.164226    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-kube-api-access-h7g25\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-kube-api-access-h7g25\") pod \"cc3e8457-0670-4daa-8354-e75592d5b17a\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") "
Nov 03 12:56:50.164377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.164300    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-plugins-config-volume\") pod \"cc3e8457-0670-4daa-8354-e75592d5b17a\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") "
Nov 03 12:56:50.164377 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.164342    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-routing-overlay-volume\") pod \"cc3e8457-0670-4daa-8354-e75592d5b17a\" (UID: \"cc3e8457-0670-4daa-8354-e75592d5b17a\") "
Nov 03 12:56:50.165297 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.165268    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-routing-overlay-volume" pod "cc3e8457-0670-4daa-8354-e75592d5b17a" (UID: "cc3e8457-0670-4daa-8354-e75592d5b17a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:56:50.165716 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.165679    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-plugins-config-volume" pod "cc3e8457-0670-4daa-8354-e75592d5b17a" (UID: "cc3e8457-0670-4daa-8354-e75592d5b17a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:56:50.167159 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.167138    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-kube-api-access-h7g25" pod "cc3e8457-0670-4daa-8354-e75592d5b17a" (UID: "cc3e8457-0670-4daa-8354-e75592d5b17a"). InnerVolumeSpecName "kube-api-access-h7g25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:56:50.167406 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.167386    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-provider-credentials" pod "cc3e8457-0670-4daa-8354-e75592d5b17a" (UID: "cc3e8457-0670-4daa-8354-e75592d5b17a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:56:50.265860 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.265825    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:56:50.265860 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.265857    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc3e8457-0670-4daa-8354-e75592d5b17a-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:56:50.266079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.265870    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:56:50.266079 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.265879    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7g25\" (UniqueName: \"kubernetes.io/projected/cc3e8457-0670-4daa-8354-e75592d5b17a-kube-api-access-h7g25\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:56:50.545725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.535066    4909 generic.go:356] "Generic (PLEG): container finished" podID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerID="5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170" exitCode=0
Nov 03 12:56:50.545725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.535804    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"
Nov 03 12:56:50.545725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.536122    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" event={"ID":"cc3e8457-0670-4daa-8354-e75592d5b17a","Type":"ContainerDied","Data":"5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170"}
Nov 03 12:56:50.545725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.536171    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9" event={"ID":"cc3e8457-0670-4daa-8354-e75592d5b17a","Type":"ContainerDied","Data":"d182e5f9aa8f968bd9eb99c6323dafbc8501b0784f1f6f549ed34e4b58269829"}
Nov 03 12:56:50.545725 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.536203    4909 scope.go:122] "RemoveContainer" containerID="5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170"
Nov 03 12:56:50.612180 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.612158    4909 scope.go:122] "RemoveContainer" containerID="5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170"
Nov 03 12:56:50.612571 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:56:50.612547    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170\": container with ID starting with 5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170 not found: ID does not exist" containerID="5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170"
Nov 03 12:56:50.612647 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.612581    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170"} err="failed to get container status \"5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170\": rpc error: code = NotFound desc = could not find container \"5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170\": container with ID starting with 5dcd7c28b5ca198496cb7a26519e674a78f8deca54c60ffd3f245f59b7629170 not found: ID does not exist"
Nov 03 12:56:50.721801 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.721770    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"]
Nov 03 12:56:50.724732 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:50.724690    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57cf66cd74-fj4g9"]
Nov 03 12:56:51.125915 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:51.125881    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" path="/var/lib/kubelet/pods/cc3e8457-0670-4daa-8354-e75592d5b17a/volumes"
Nov 03 12:56:55.810036 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.809987    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"]
Nov 03 12:56:55.811553 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.811511    4909 cpu_manager.go:414] "containerMap: removing container" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing"
Nov 03 12:56:55.811726 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.811693    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing"
Nov 03 12:56:55.812058 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.812043    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3e8457-0670-4daa-8354-e75592d5b17a" containerName="payload-processing"
Nov 03 12:56:55.824729 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.819283    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:55.863435 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.863213    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"]
Nov 03 12:56:55.899030 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.898996    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-532442\""
Nov 03 12:56:55.899433 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.899412    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-532442-dockercfg-wgc24\""
Nov 03 12:56:55.902307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.902113    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36097829-e13a-4805-9a9e-e020c0b2fa4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:55.902307 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:55.902211    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8557\" (UniqueName: \"kubernetes.io/projected/36097829-e13a-4805-9a9e-e020c0b2fa4c-kube-api-access-l8557\") pod \"payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:56.003259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:56.003228    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36097829-e13a-4805-9a9e-e020c0b2fa4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:56.003431 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:56.003309    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8557\" (UniqueName: \"kubernetes.io/projected/36097829-e13a-4805-9a9e-e020c0b2fa4c-kube-api-access-l8557\") pod \"payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:56.004011 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:56.003989    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36097829-e13a-4805-9a9e-e020c0b2fa4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:56.057014 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:56.056981    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8557\" (UniqueName: \"kubernetes.io/projected/36097829-e13a-4805-9a9e-e020c0b2fa4c-kube-api-access-l8557\") pod \"payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:56.170123 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:56.170047    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:56.542359 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:56.542330    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"]
Nov 03 12:56:56.544897 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:56:56.544860    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36097829_e13a_4805_9a9e_e020c0b2fa4c.slice/crio-5eae79e18f03adb99b4d27c60a331d074efec0a6758243533f48e9ef6bde7db7 WatchSource:0}: Error finding container 5eae79e18f03adb99b4d27c60a331d074efec0a6758243533f48e9ef6bde7db7: Status 404 returned error can't find the container with id 5eae79e18f03adb99b4d27c60a331d074efec0a6758243533f48e9ef6bde7db7
Nov 03 12:56:56.621832 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:56.621804    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l" event={"ID":"36097829-e13a-4805-9a9e-e020c0b2fa4c","Type":"ContainerStarted","Data":"5eae79e18f03adb99b4d27c60a331d074efec0a6758243533f48e9ef6bde7db7"}
Nov 03 12:56:57.228898 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:57.228855    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:56:57.229342 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:57.228930    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:56:57.713861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:57.699642    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l" event={"ID":"36097829-e13a-4805-9a9e-e020c0b2fa4c","Type":"ContainerStarted","Data":"a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84"}
Nov 03 12:56:57.713861 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:57.713788    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:56:57.820153 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:57.820096    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l" podStartSLOduration=2.820078386 podStartE2EDuration="2.820078386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:56:57.819596662 +0000 UTC m=+3966.832063342" watchObservedRunningTime="2026-11-03 12:56:57.820078386 +0000 UTC m=+3966.832545067"
Nov 03 12:56:59.736105 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:59.736073    4909 generic.go:356] "Generic (PLEG): container finished" podID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerID="82842d7e672a6524c760ca88d152b7453aabc92e8170c64a603f84ce29420a06" exitCode=0
Nov 03 12:56:59.736624 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:59.736150    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" event={"ID":"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2","Type":"ContainerDied","Data":"82842d7e672a6524c760ca88d152b7453aabc92e8170c64a603f84ce29420a06"}
Nov 03 12:56:59.736888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:59.736869    4909 scope.go:122] "RemoveContainer" containerID="82842d7e672a6524c760ca88d152b7453aabc92e8170c64a603f84ce29420a06"
Nov 03 12:56:59.738504 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:59.738473    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:56:59.738625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:59.738556    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:56:59.738625 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:56:59.738602    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:57:00.771172 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:00.771136    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" event={"ID":"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2","Type":"ContainerStarted","Data":"0e80d0f24b4815a6784abbc1cd9d92d4125e5ce5d00e942e444f83aa4bda89a7"}
Nov 03 12:57:01.599735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:01.599686    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:57:01.599909 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:01.599885    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 12:57:01.599971 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:01.599937    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 12:57:02.599096 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:02.599062    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 12:57:02.599520 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:02.599112    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 12:57:03.599487 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:03.599449    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 12:57:03.599929 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:03.599506    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 12:57:04.599048 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:04.599010    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 12:57:04.599233 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:04.599073    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 12:57:04.698883 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:04.698843    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:57:05.217719 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.217328    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"]
Nov 03 12:57:05.236912 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.236881    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.287841 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.287810    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"]
Nov 03 12:57:05.319717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.315005    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fb74b5-dockercfg-j5bjf\""
Nov 03 12:57:05.319717 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.315100    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fb74b5\""
Nov 03 12:57:05.439724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.436921    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca135c32-5abe-4295-8985-990029dfa602-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.439724 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.437059    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2z4\" (UniqueName: \"kubernetes.io/projected/ca135c32-5abe-4295-8985-990029dfa602-kube-api-access-2h2z4\") pod \"payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.505976 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.505903    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"]
Nov 03 12:57:05.510490 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.510469    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"]
Nov 03 12:57:05.510644 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.510587    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.562902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.552456    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2z4\" (UniqueName: \"kubernetes.io/projected/ca135c32-5abe-4295-8985-990029dfa602-kube-api-access-2h2z4\") pod \"payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.562902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.552586    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gwn\" (UniqueName: \"kubernetes.io/projected/b91c14c7-3143-4427-a746-6758a9ac1170-kube-api-access-q2gwn\") pod \"payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.562902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.552731    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca135c32-5abe-4295-8985-990029dfa602-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.562902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.552816    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91c14c7-3143-4427-a746-6758a9ac1170-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.562902 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.554078    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca135c32-5abe-4295-8985-990029dfa602-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.599657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.599625    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 12:57:05.599798 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.599675    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 12:57:05.605836 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.605805    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2z4\" (UniqueName: \"kubernetes.io/projected/ca135c32-5abe-4295-8985-990029dfa602-kube-api-access-2h2z4\") pod \"payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.662747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.661478    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gwn\" (UniqueName: \"kubernetes.io/projected/b91c14c7-3143-4427-a746-6758a9ac1170-kube-api-access-q2gwn\") pod \"payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.662747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.661682    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91c14c7-3143-4427-a746-6758a9ac1170-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.663747 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.663690    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91c14c7-3143-4427-a746-6758a9ac1170-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.715231 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.715204    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gwn\" (UniqueName: \"kubernetes.io/projected/b91c14c7-3143-4427-a746-6758a9ac1170-kube-api-access-q2gwn\") pod \"payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.825185 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.825101    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:05.896061 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:05.896029    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:05.967800 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:57:05.967006    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:57:05.967800 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:57:05.967050    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6e306ea651ee: secret "agc-cred-6e306ea651ee" not found
Nov 03 12:57:05.967800 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:57:05.967069    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: [secret "agc-cred-1ab9c770bc22" not found, secret "agc-cred-6e306ea651ee" not found]
Nov 03 12:57:05.967800 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:57:05.967128    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 12:59:07.967109437 +0000 UTC m=+4096.979576094 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : [secret "agc-cred-1ab9c770bc22" not found, secret "agc-cred-6e306ea651ee" not found]
Nov 03 12:57:06.222979 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.222945    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"]
Nov 03 12:57:06.248129 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:57:06.248089    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91c14c7_3143_4427_a746_6758a9ac1170.slice/crio-a4e3666a57a009e49a3b5544e6eb6a034b440fbb3eda90c808a75ee4f2dabeb3 WatchSource:0}: Error finding container a4e3666a57a009e49a3b5544e6eb6a034b440fbb3eda90c808a75ee4f2dabeb3: Status 404 returned error can't find the container with id a4e3666a57a009e49a3b5544e6eb6a034b440fbb3eda90c808a75ee4f2dabeb3
Nov 03 12:57:06.333590 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:57:06.333560    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca135c32_5abe_4295_8985_990029dfa602.slice/crio-6d7a1f9bee631a0b85a4feed3409e069d54f90b2976d49411eb1ec3028933879 WatchSource:0}: Error finding container 6d7a1f9bee631a0b85a4feed3409e069d54f90b2976d49411eb1ec3028933879: Status 404 returned error can't find the container with id 6d7a1f9bee631a0b85a4feed3409e069d54f90b2976d49411eb1ec3028933879
Nov 03 12:57:06.334584 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.334557    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"]
Nov 03 12:57:06.604888 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.604792    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:57:06.605248 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.605208    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:57:06.606009 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.605982    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 12:57:06.816384 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.816346    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59" event={"ID":"b91c14c7-3143-4427-a746-6758a9ac1170","Type":"ContainerStarted","Data":"f02a9661d6d32a80fed935b26ca4a50824d0ff8f2a0e52cf6782db45b2af7a37"}
Nov 03 12:57:06.816950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.816409    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59" event={"ID":"b91c14c7-3143-4427-a746-6758a9ac1170","Type":"ContainerStarted","Data":"a4e3666a57a009e49a3b5544e6eb6a034b440fbb3eda90c808a75ee4f2dabeb3"}
Nov 03 12:57:06.816950 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.816477    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:06.818523 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.818491    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl" event={"ID":"ca135c32-5abe-4295-8985-990029dfa602","Type":"ContainerStarted","Data":"79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888"}
Nov 03 12:57:06.818666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.818550    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl" event={"ID":"ca135c32-5abe-4295-8985-990029dfa602","Type":"ContainerStarted","Data":"6d7a1f9bee631a0b85a4feed3409e069d54f90b2976d49411eb1ec3028933879"}
Nov 03 12:57:06.818666 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.818654    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:06.972074 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:06.972006    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl" podStartSLOduration=1.971986808 podStartE2EDuration="1.971986808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:57:06.969357618 +0000 UTC m=+3975.981824280" watchObservedRunningTime="2026-11-03 12:57:06.971986808 +0000 UTC m=+3975.984453472"
Nov 03 12:57:07.075071 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:07.075013    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59" podStartSLOduration=2.074995259 podStartE2EDuration="2.074995259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:57:07.074205226 +0000 UTC m=+3976.086671895" watchObservedRunningTime="2026-11-03 12:57:07.074995259 +0000 UTC m=+3976.087461924"
Nov 03 12:57:08.994723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:08.992794    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:57:08.994723 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:08.992874    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.010721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:08.995810    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:57:09.010721 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:08.995985    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.565132 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:57:09.565099    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:57:09.739735 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:09.739688    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 12:57:17.828563 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:17.828532    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 12:57:17.829062 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:17.828597    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 12:57:39.564645 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:57:39.564564    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:57:44.851833 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:44.851801    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"]
Nov 03 12:57:44.857789 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:44.857766    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:44.929567 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:44.929525    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"]
Nov 03 12:57:44.975627 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:44.944801    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65d7\" (UniqueName: \"kubernetes.io/projected/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-kube-api-access-m65d7\") pod \"payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:44.976137 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:44.976104    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:44.976555 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:44.967239    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-aa9221-dockercfg-x4gmf\""
Nov 03 12:57:44.976718 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:44.967089    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-aa9221\""
Nov 03 12:57:45.090680 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:45.090645    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65d7\" (UniqueName: \"kubernetes.io/projected/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-kube-api-access-m65d7\") pod \"payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:45.090680 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:45.090716    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:45.091952 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:45.091876    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:45.176869 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:45.176732    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65d7\" (UniqueName: \"kubernetes.io/projected/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-kube-api-access-m65d7\") pod \"payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:45.475127 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:45.475093    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:45.844690 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:45.844665    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"]
Nov 03 12:57:45.845898 ip-10-0-131-136 kubenswrapper[4909]: W1103 12:57:45.845871    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3b4dd9_b21e_4f1b_bc82_517349e20f12.slice/crio-f7cfbd5dc62497ab77cf479db446100ee0ddb60e43ec51e24e87d17e79bc3bd3 WatchSource:0}: Error finding container f7cfbd5dc62497ab77cf479db446100ee0ddb60e43ec51e24e87d17e79bc3bd3: Status 404 returned error can't find the container with id f7cfbd5dc62497ab77cf479db446100ee0ddb60e43ec51e24e87d17e79bc3bd3
Nov 03 12:57:46.115822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:46.115742    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q" event={"ID":"8d3b4dd9-b21e-4f1b-bc82-517349e20f12","Type":"ContainerStarted","Data":"14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c"}
Nov 03 12:57:46.115822 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:46.115804    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q" event={"ID":"8d3b4dd9-b21e-4f1b-bc82-517349e20f12","Type":"ContainerStarted","Data":"f7cfbd5dc62497ab77cf479db446100ee0ddb60e43ec51e24e87d17e79bc3bd3"}
Nov 03 12:57:46.116215 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:46.115832    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:57:46.222596 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:46.222546    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q" podStartSLOduration=2.222531866 podStartE2EDuration="2.222531866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:57:46.219752975 +0000 UTC m=+4015.232219639" watchObservedRunningTime="2026-11-03 12:57:46.222531866 +0000 UTC m=+4015.234998528"
Nov 03 12:57:57.127447 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:57:57.127419    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 12:58:09.564911 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:58:09.564878    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:58:28.157623 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:28.157590    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:58:39.484564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:39.484534    4909 generic.go:356] "Generic (PLEG): container finished" podID="b73f9e32-8924-4f74-8623-00be96ff490b" containerID="8d8e9a4dbf3861a82d056dc853655b380f67017aede09dab51e75dd9de84161a" exitCode=0
Nov 03 12:58:39.485085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:39.484613    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" event={"ID":"b73f9e32-8924-4f74-8623-00be96ff490b","Type":"ContainerDied","Data":"8d8e9a4dbf3861a82d056dc853655b380f67017aede09dab51e75dd9de84161a"}
Nov 03 12:58:39.485085 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:39.484988    4909 scope.go:122] "RemoveContainer" containerID="8d8e9a4dbf3861a82d056dc853655b380f67017aede09dab51e75dd9de84161a"
Nov 03 12:58:39.486564 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:39.486537    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:58:39.486657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:39.486602    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:58:39.486657 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:39.486634    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 12:58:39.566277 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:58:39.566243    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:58:40.493444 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:40.493408    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" event={"ID":"b73f9e32-8924-4f74-8623-00be96ff490b","Type":"ContainerStarted","Data":"f1f7e3e2158dbbc1a756c5eeb004d311471ad1039f1456c1a1363a5ee357f038"}
Nov 03 12:58:40.871951 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:40.871869    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:58:40.872106 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:40.872075    4909 patch_prober.go:29] interesting pod/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 12:58:40.872169 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:40.872119    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 12:58:41.872238 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:41.872200    4909 patch_prober.go:29] interesting pod/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 12:58:41.872745 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:41.872277    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 12:58:42.872477 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:42.872442    4909 patch_prober.go:29] interesting pod/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 12:58:42.872931 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:42.872505    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 12:58:43.872259 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:43.872216    4909 patch_prober.go:29] interesting pod/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 12:58:43.872457 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:43.872281    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 12:58:44.872561 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:44.872521    4909 patch_prober.go:29] interesting pod/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 12:58:44.873024 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:44.872590    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 12:58:45.872865 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:45.872819    4909 patch_prober.go:29] interesting pod/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 12:58:45.873296 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:45.872887    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 12:58:46.877053 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:46.877022    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:58:46.877522 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:46.877505    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:58:46.878147 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:58:46.878128    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 12:59:03.357635 ip-10-0-131-136 kubenswrapper[4909]: I1103 12:59:03.357546    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 12:59:07.994091 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:59:07.994055    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 12:59:07.994091 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:59:07.994081    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6e306ea651ee: secret "agc-cred-6e306ea651ee" not found
Nov 03 12:59:07.994091 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:59:07.994094    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: [secret "agc-cred-1ab9c770bc22" not found, secret "agc-cred-6e306ea651ee" not found]
Nov 03 12:59:07.994661 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:59:07.994152    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 13:01:09.994138768 +0000 UTC m=+4219.006605426 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : [secret "agc-cred-1ab9c770bc22" not found, secret "agc-cred-6e306ea651ee" not found]
Nov 03 12:59:09.565082 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:59:09.565046    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 12:59:39.564242 ip-10-0-131-136 kubenswrapper[4909]: E1103 12:59:39.564197    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:00:00.182576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.182540    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"]
Nov 03 13:00:00.198980 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.198843    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"]
Nov 03 13:00:00.199164 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.199046    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.204108 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.204084    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"]
Nov 03 13:00:00.204228 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.204205    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.208604 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.208585    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.228427 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.228406    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"]
Nov 03 13:00:00.232297 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.232279    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"]
Nov 03 13:00:00.232380 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.232366    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.236337 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.236323    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"]
Nov 03 13:00:00.236437 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.236420    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.240057 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.240036    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"]
Nov 03 13:00:00.240151 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.240136    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.244113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.244096    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"]
Nov 03 13:00:00.244193 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.244155    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.248846 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.248825    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.305751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.305725    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f243926b-8892-434a-afed-9e62b602c3ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.305751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.305769    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01030a13-b2a2-423b-aa1e-bf92c969edbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.306182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.305822    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tv66\" (UniqueName: \"kubernetes.io/projected/5fab2913-3757-4544-b72e-8219fe9dac4d-kube-api-access-8tv66\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.306182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.305870    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5660732-c985-4aff-ae7f-a019ca46e893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.306182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.305894    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/9602ddec-e761-4948-95f0-96427993446d-kube-api-access-ptccw\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.306182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.305940    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8m8\" (UniqueName: \"kubernetes.io/projected/a5660732-c985-4aff-ae7f-a019ca46e893-kube-api-access-tp8m8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.306182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306161    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fab2913-3757-4544-b72e-8219fe9dac4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.306182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306183    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9602ddec-e761-4948-95f0-96427993446d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.306567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306208    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqjv\" (UniqueName: \"kubernetes.io/projected/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-kube-api-access-qtqjv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.306567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306244    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.306567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306308    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzc98\" (UniqueName: \"kubernetes.io/projected/4b8e61c0-6280-462c-9d6b-95665790b662-kube-api-access-mzc98\") pod \"maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.306567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306334    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn9p\" (UniqueName: \"kubernetes.io/projected/01030a13-b2a2-423b-aa1e-bf92c969edbd-kube-api-access-pmn9p\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.306567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306362    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a5c80b-3ed7-4009-8c26-bc262fb07651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.306567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306460    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e61c0-6280-462c-9d6b-95665790b662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.306567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306522    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv6bf\" (UniqueName: \"kubernetes.io/projected/f243926b-8892-434a-afed-9e62b602c3ca-kube-api-access-hv6bf\") pod \"maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.306987 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.306574    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/44a5c80b-3ed7-4009-8c26-bc262fb07651-kube-api-access-8qxsx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.311244 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.311222    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"]
Nov 03 13:00:00.321437 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.321419    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:00:00.328588 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.328565    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"]
Nov 03 13:00:00.332867 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.332847    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"]
Nov 03 13:00:00.357526 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.356903    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"]
Nov 03 13:00:00.362339 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.362315    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"]
Nov 03 13:00:00.365739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.365610    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"]
Nov 03 13:00:00.369304 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.368876    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"]
Nov 03 13:00:00.402798 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.402774    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"]
Nov 03 13:00:00.407984 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.407952    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc98\" (UniqueName: \"kubernetes.io/projected/4b8e61c0-6280-462c-9d6b-95665790b662-kube-api-access-mzc98\") pod \"maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.408140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408007    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn9p\" (UniqueName: \"kubernetes.io/projected/01030a13-b2a2-423b-aa1e-bf92c969edbd-kube-api-access-pmn9p\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.408140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408053    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a5c80b-3ed7-4009-8c26-bc262fb07651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.408140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408106    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e61c0-6280-462c-9d6b-95665790b662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.408410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408144    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv6bf\" (UniqueName: \"kubernetes.io/projected/f243926b-8892-434a-afed-9e62b602c3ca-kube-api-access-hv6bf\") pod \"maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.408410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/44a5c80b-3ed7-4009-8c26-bc262fb07651-kube-api-access-8qxsx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.408410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408270    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f243926b-8892-434a-afed-9e62b602c3ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.408410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408314    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01030a13-b2a2-423b-aa1e-bf92c969edbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.408410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408363    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tv66\" (UniqueName: \"kubernetes.io/projected/5fab2913-3757-4544-b72e-8219fe9dac4d-kube-api-access-8tv66\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.409307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408420    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5660732-c985-4aff-ae7f-a019ca46e893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.409307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408445    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/9602ddec-e761-4948-95f0-96427993446d-kube-api-access-ptccw\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.409307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408499    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8m8\" (UniqueName: \"kubernetes.io/projected/a5660732-c985-4aff-ae7f-a019ca46e893-kube-api-access-tp8m8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.409307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408569    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fab2913-3757-4544-b72e-8219fe9dac4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.409307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408595    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9602ddec-e761-4948-95f0-96427993446d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.409307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408625    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqjv\" (UniqueName: \"kubernetes.io/projected/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-kube-api-access-qtqjv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.409307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.408672    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.409890 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.409423    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f243926b-8892-434a-afed-9e62b602c3ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.409890 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.409505    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.410234 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.410210    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01030a13-b2a2-423b-aa1e-bf92c969edbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.410630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.410604    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a5c80b-3ed7-4009-8c26-bc262fb07651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.411171 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.411147    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5660732-c985-4aff-ae7f-a019ca46e893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.411330 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.411305    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e61c0-6280-462c-9d6b-95665790b662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.412520 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.412492    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fab2913-3757-4544-b72e-8219fe9dac4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.412630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.412522    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9602ddec-e761-4948-95f0-96427993446d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.463139 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.463108    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzc98\" (UniqueName: \"kubernetes.io/projected/4b8e61c0-6280-462c-9d6b-95665790b662-kube-api-access-mzc98\") pod \"maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.463296 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.463275    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn9p\" (UniqueName: \"kubernetes.io/projected/01030a13-b2a2-423b-aa1e-bf92c969edbd-kube-api-access-pmn9p\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.463572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.463546    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tv66\" (UniqueName: \"kubernetes.io/projected/5fab2913-3757-4544-b72e-8219fe9dac4d-kube-api-access-8tv66\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.464505 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.464474    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8m8\" (UniqueName: \"kubernetes.io/projected/a5660732-c985-4aff-ae7f-a019ca46e893-kube-api-access-tp8m8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.464710 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.464679    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv6bf\" (UniqueName: \"kubernetes.io/projected/f243926b-8892-434a-afed-9e62b602c3ca-kube-api-access-hv6bf\") pod \"maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.465052 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.465027    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/9602ddec-e761-4948-95f0-96427993446d-kube-api-access-ptccw\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.465534 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.465510    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/44a5c80b-3ed7-4009-8c26-bc262fb07651-kube-api-access-8qxsx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.465615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.465601    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqjv\" (UniqueName: \"kubernetes.io/projected/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-kube-api-access-qtqjv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.515225 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.515201    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:00.522225 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.522206    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:00.528883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.528858    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:00.549561 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.549538    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:00.559003 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.558977    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:00.566661 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.566628    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:00.577334 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.576886    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:00.581383 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.581331    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:00.769898 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.769873    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"]
Nov 03 13:00:00.899426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:00.899399    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"]
Nov 03 13:00:00.900259 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:00:00.900228    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5660732_c985_4aff_ae7f_a019ca46e893.slice/crio-2d2c5a817bdc599f2cb45be38f60dbe9b8f4ca58e528e71da0e43d08ce8dca9c WatchSource:0}: Error finding container 2d2c5a817bdc599f2cb45be38f60dbe9b8f4ca58e528e71da0e43d08ce8dca9c: Status 404 returned error can't find the container with id 2d2c5a817bdc599f2cb45be38f60dbe9b8f4ca58e528e71da0e43d08ce8dca9c
Nov 03 13:00:01.045275 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.045249    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"]
Nov 03 13:00:01.065116 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.065082    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" event={"ID":"a5660732-c985-4aff-ae7f-a019ca46e893","Type":"ContainerStarted","Data":"2d2c5a817bdc599f2cb45be38f60dbe9b8f4ca58e528e71da0e43d08ce8dca9c"}
Nov 03 13:00:01.086383 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.086358    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"]
Nov 03 13:00:01.114042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.114010    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"]
Nov 03 13:00:01.124884 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.124861    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"]
Nov 03 13:00:01.141919 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:00:01.141890    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8e61c0_6280_462c_9d6b_95665790b662.slice/crio-6cb9c460a5ab0d2c0707b908d317d699d67267be3d415b13a1779735d9fbda29 WatchSource:0}: Error finding container 6cb9c460a5ab0d2c0707b908d317d699d67267be3d415b13a1779735d9fbda29: Status 404 returned error can't find the container with id 6cb9c460a5ab0d2c0707b908d317d699d67267be3d415b13a1779735d9fbda29
Nov 03 13:00:01.143120 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:00:01.143095    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01030a13_b2a2_423b_aa1e_bf92c969edbd.slice/crio-3742b7d580bbb085827302fcbd86d0f5d62a0e273e9b458dfc6c3fcd8b8a52e2 WatchSource:0}: Error finding container 3742b7d580bbb085827302fcbd86d0f5d62a0e273e9b458dfc6c3fcd8b8a52e2: Status 404 returned error can't find the container with id 3742b7d580bbb085827302fcbd86d0f5d62a0e273e9b458dfc6c3fcd8b8a52e2
Nov 03 13:00:01.144567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.144550    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"]
Nov 03 13:00:01.145530 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:00:01.145507    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a5c80b_3ed7_4009_8c26_bc262fb07651.slice/crio-5d4c16836094f68a2c161205ecf2dfbf8adeb6a47a259f3c78bc086552daf502 WatchSource:0}: Error finding container 5d4c16836094f68a2c161205ecf2dfbf8adeb6a47a259f3c78bc086552daf502: Status 404 returned error can't find the container with id 5d4c16836094f68a2c161205ecf2dfbf8adeb6a47a259f3c78bc086552daf502
Nov 03 13:00:01.155609 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:00:01.155578    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fab2913_3757_4544_b72e_8219fe9dac4d.slice/crio-870da8da4a596a3a1be3eb0d67ba757c28413685a9168d4f25f98e2333e97b68 WatchSource:0}: Error finding container 870da8da4a596a3a1be3eb0d67ba757c28413685a9168d4f25f98e2333e97b68: Status 404 returned error can't find the container with id 870da8da4a596a3a1be3eb0d67ba757c28413685a9168d4f25f98e2333e97b68
Nov 03 13:00:01.198753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.198724    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"]
Nov 03 13:00:01.235838 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:01.235811    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"]
Nov 03 13:00:01.236768 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:00:01.236738    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf243926b_8892_434a_afed_9e62b602c3ca.slice/crio-711aad28520e8276d03b43a54e748debc6082ea8b79dd28eebb30c15160dc7b2 WatchSource:0}: Error finding container 711aad28520e8276d03b43a54e748debc6082ea8b79dd28eebb30c15160dc7b2: Status 404 returned error can't find the container with id 711aad28520e8276d03b43a54e748debc6082ea8b79dd28eebb30c15160dc7b2
Nov 03 13:00:02.080042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.080001    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" event={"ID":"44a5c80b-3ed7-4009-8c26-bc262fb07651","Type":"ContainerStarted","Data":"59318f543907b0a317cd30554a811ba23daad7c7124beed57c9f65d084e2ffc5"}
Nov 03 13:00:02.081526 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.081495    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" event={"ID":"44a5c80b-3ed7-4009-8c26-bc262fb07651","Type":"ContainerStarted","Data":"5d4c16836094f68a2c161205ecf2dfbf8adeb6a47a259f3c78bc086552daf502"}
Nov 03 13:00:02.083932 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.083908    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" event={"ID":"01030a13-b2a2-423b-aa1e-bf92c969edbd","Type":"ContainerStarted","Data":"40f47979f6fffbe5d0d3ddcec72e2705a3567afc0d9bc6b2ef948dfd3b2c1596"}
Nov 03 13:00:02.084121 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.084102    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" event={"ID":"01030a13-b2a2-423b-aa1e-bf92c969edbd","Type":"ContainerStarted","Data":"3742b7d580bbb085827302fcbd86d0f5d62a0e273e9b458dfc6c3fcd8b8a52e2"}
Nov 03 13:00:02.097641 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.097610    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" event={"ID":"4b8e61c0-6280-462c-9d6b-95665790b662","Type":"ContainerStarted","Data":"93ea2ebbdbaecf1386daacfd62559d0d34eae7a5292cbce26a3ae9bad786007e"}
Nov 03 13:00:02.097953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.097929    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" event={"ID":"4b8e61c0-6280-462c-9d6b-95665790b662","Type":"ContainerStarted","Data":"6cb9c460a5ab0d2c0707b908d317d699d67267be3d415b13a1779735d9fbda29"}
Nov 03 13:00:02.161743 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.159564    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" event={"ID":"79bbbd2b-c562-4575-9e20-6bf81c2ce99e","Type":"ContainerStarted","Data":"733c4c833bf85f2f0b105b48a095f26f2730d9d7e125194d54d0af8a1d482804"}
Nov 03 13:00:02.161743 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.159631    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" event={"ID":"79bbbd2b-c562-4575-9e20-6bf81c2ce99e","Type":"ContainerStarted","Data":"319318051d31cd76460739048cec59f8a0503e2c804e767ab277519b19d27670"}
Nov 03 13:00:02.204271 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.204171    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" event={"ID":"9602ddec-e761-4948-95f0-96427993446d","Type":"ContainerStarted","Data":"381ec8939dd07679fe4b170940d476dcf321a06acd9d12b4f10980e4cfe98e44"}
Nov 03 13:00:02.206687 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.205111    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" event={"ID":"9602ddec-e761-4948-95f0-96427993446d","Type":"ContainerStarted","Data":"276a0ee26eec442e1bbb67b04396df0c37f77b64ee16a0c3b929effd9c0b5c29"}
Nov 03 13:00:02.209562 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.209539    4909 generic.go:356] "Generic (PLEG): container finished" podID="a5660732-c985-4aff-ae7f-a019ca46e893" containerID="9d4dc9c28cd78bba80afe2f97579f086930a13b4df1d3462134d5affe79bbb6b" exitCode=60
Nov 03 13:00:02.210203 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.210184    4909 scope.go:122] "RemoveContainer" containerID="9d4dc9c28cd78bba80afe2f97579f086930a13b4df1d3462134d5affe79bbb6b"
Nov 03 13:00:02.210778 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.210756    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" event={"ID":"a5660732-c985-4aff-ae7f-a019ca46e893","Type":"ContainerDied","Data":"9d4dc9c28cd78bba80afe2f97579f086930a13b4df1d3462134d5affe79bbb6b"}
Nov 03 13:00:02.225012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.223275    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" event={"ID":"5fab2913-3757-4544-b72e-8219fe9dac4d","Type":"ContainerStarted","Data":"dd7962686955368736c00373afe0f033b547c418590cb44b93f4cad9f53621a3"}
Nov 03 13:00:02.225012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.223343    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" event={"ID":"5fab2913-3757-4544-b72e-8219fe9dac4d","Type":"ContainerStarted","Data":"870da8da4a596a3a1be3eb0d67ba757c28413685a9168d4f25f98e2333e97b68"}
Nov 03 13:00:02.228970 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.228905    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" podStartSLOduration=2.228889559 podStartE2EDuration="2.228889559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.218000397 +0000 UTC m=+4151.230467061" watchObservedRunningTime="2026-11-03 13:00:02.228889559 +0000 UTC m=+4151.241356223"
Nov 03 13:00:02.234502 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.234474    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" event={"ID":"f243926b-8892-434a-afed-9e62b602c3ca","Type":"ContainerStarted","Data":"b118def65225af7f9db3a23cf610c0f055b6ea0a0891c63a193d38b393d2a62b"}
Nov 03 13:00:02.234683 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.234659    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" event={"ID":"f243926b-8892-434a-afed-9e62b602c3ca","Type":"ContainerStarted","Data":"711aad28520e8276d03b43a54e748debc6082ea8b79dd28eebb30c15160dc7b2"}
Nov 03 13:00:02.348815 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.347935    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:00:02.431755 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.431671    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" podStartSLOduration=2.431650825 podStartE2EDuration="2.431650825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.430258201 +0000 UTC m=+4151.442724867" watchObservedRunningTime="2026-11-03 13:00:02.431650825 +0000 UTC m=+4151.444117487"
Nov 03 13:00:02.534878 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.534810    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" podStartSLOduration=2.534791394 podStartE2EDuration="2.534791394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.533750736 +0000 UTC m=+4151.546217400" watchObservedRunningTime="2026-11-03 13:00:02.534791394 +0000 UTC m=+4151.547258058"
Nov 03 13:00:02.640774 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.640727    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" podStartSLOduration=2.64069253 podStartE2EDuration="2.64069253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.637134013 +0000 UTC m=+4151.649600675" watchObservedRunningTime="2026-11-03 13:00:02.64069253 +0000 UTC m=+4151.653159193"
Nov 03 13:00:02.741827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.741781    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" podStartSLOduration=2.741768258 podStartE2EDuration="2.741768258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.740626124 +0000 UTC m=+4151.753092788" watchObservedRunningTime="2026-11-03 13:00:02.741768258 +0000 UTC m=+4151.754234938"
Nov 03 13:00:02.849318 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.849268    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" podStartSLOduration=2.849249944 podStartE2EDuration="2.849249944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.844825067 +0000 UTC m=+4151.857291729" watchObservedRunningTime="2026-11-03 13:00:02.849249944 +0000 UTC m=+4151.861716607"
Nov 03 13:00:02.957783 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:02.957086    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" podStartSLOduration=2.957067986 podStartE2EDuration="2.957067986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.952993803 +0000 UTC m=+4151.965460467" watchObservedRunningTime="2026-11-03 13:00:02.957067986 +0000 UTC m=+4151.969534650"
Nov 03 13:00:03.248755 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.248598    4909 generic.go:356] "Generic (PLEG): container finished" podID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerID="59318f543907b0a317cd30554a811ba23daad7c7124beed57c9f65d084e2ffc5" exitCode=60
Nov 03 13:00:03.249394 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.248848    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" event={"ID":"44a5c80b-3ed7-4009-8c26-bc262fb07651","Type":"ContainerDied","Data":"59318f543907b0a317cd30554a811ba23daad7c7124beed57c9f65d084e2ffc5"}
Nov 03 13:00:03.249394 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.249124    4909 scope.go:122] "RemoveContainer" containerID="59318f543907b0a317cd30554a811ba23daad7c7124beed57c9f65d084e2ffc5"
Nov 03 13:00:03.254473 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.254444    4909 generic.go:356] "Generic (PLEG): container finished" podID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerID="40f47979f6fffbe5d0d3ddcec72e2705a3567afc0d9bc6b2ef948dfd3b2c1596" exitCode=60
Nov 03 13:00:03.254611 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.254526    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" event={"ID":"01030a13-b2a2-423b-aa1e-bf92c969edbd","Type":"ContainerDied","Data":"40f47979f6fffbe5d0d3ddcec72e2705a3567afc0d9bc6b2ef948dfd3b2c1596"}
Nov 03 13:00:03.255059 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.255031    4909 scope.go:122] "RemoveContainer" containerID="40f47979f6fffbe5d0d3ddcec72e2705a3567afc0d9bc6b2ef948dfd3b2c1596"
Nov 03 13:00:03.257422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.257397    4909 generic.go:356] "Generic (PLEG): container finished" podID="4b8e61c0-6280-462c-9d6b-95665790b662" containerID="93ea2ebbdbaecf1386daacfd62559d0d34eae7a5292cbce26a3ae9bad786007e" exitCode=60
Nov 03 13:00:03.257672 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.257640    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" event={"ID":"4b8e61c0-6280-462c-9d6b-95665790b662","Type":"ContainerDied","Data":"93ea2ebbdbaecf1386daacfd62559d0d34eae7a5292cbce26a3ae9bad786007e"}
Nov 03 13:00:03.257821 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.257779    4909 scope.go:122] "RemoveContainer" containerID="93ea2ebbdbaecf1386daacfd62559d0d34eae7a5292cbce26a3ae9bad786007e"
Nov 03 13:00:03.259395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.259374    4909 generic.go:356] "Generic (PLEG): container finished" podID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerID="733c4c833bf85f2f0b105b48a095f26f2730d9d7e125194d54d0af8a1d482804" exitCode=60
Nov 03 13:00:03.259489 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.259447    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" event={"ID":"79bbbd2b-c562-4575-9e20-6bf81c2ce99e","Type":"ContainerDied","Data":"733c4c833bf85f2f0b105b48a095f26f2730d9d7e125194d54d0af8a1d482804"}
Nov 03 13:00:03.259654 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.259636    4909 scope.go:122] "RemoveContainer" containerID="733c4c833bf85f2f0b105b48a095f26f2730d9d7e125194d54d0af8a1d482804"
Nov 03 13:00:03.261498 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.261481    4909 generic.go:356] "Generic (PLEG): container finished" podID="9602ddec-e761-4948-95f0-96427993446d" containerID="381ec8939dd07679fe4b170940d476dcf321a06acd9d12b4f10980e4cfe98e44" exitCode=60
Nov 03 13:00:03.261591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.261522    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" event={"ID":"9602ddec-e761-4948-95f0-96427993446d","Type":"ContainerDied","Data":"381ec8939dd07679fe4b170940d476dcf321a06acd9d12b4f10980e4cfe98e44"}
Nov 03 13:00:03.261814 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.261795    4909 scope.go:122] "RemoveContainer" containerID="381ec8939dd07679fe4b170940d476dcf321a06acd9d12b4f10980e4cfe98e44"
Nov 03 13:00:03.267730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.264527    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" event={"ID":"a5660732-c985-4aff-ae7f-a019ca46e893","Type":"ContainerStarted","Data":"8ff6f7bb76dda8c48f691da7939549ae306411f2a47f6197e700b56251004fdd"}
Nov 03 13:00:03.267730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.266809    4909 generic.go:356] "Generic (PLEG): container finished" podID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerID="dd7962686955368736c00373afe0f033b547c418590cb44b93f4cad9f53621a3" exitCode=60
Nov 03 13:00:03.267730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.266877    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" event={"ID":"5fab2913-3757-4544-b72e-8219fe9dac4d","Type":"ContainerDied","Data":"dd7962686955368736c00373afe0f033b547c418590cb44b93f4cad9f53621a3"}
Nov 03 13:00:03.267730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.267096    4909 scope.go:122] "RemoveContainer" containerID="dd7962686955368736c00373afe0f033b547c418590cb44b93f4cad9f53621a3"
Nov 03 13:00:03.268512 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.268281    4909 generic.go:356] "Generic (PLEG): container finished" podID="f243926b-8892-434a-afed-9e62b602c3ca" containerID="b118def65225af7f9db3a23cf610c0f055b6ea0a0891c63a193d38b393d2a62b" exitCode=60
Nov 03 13:00:03.268512 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.268300    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" event={"ID":"f243926b-8892-434a-afed-9e62b602c3ca","Type":"ContainerDied","Data":"b118def65225af7f9db3a23cf610c0f055b6ea0a0891c63a193d38b393d2a62b"}
Nov 03 13:00:03.268512 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.268476    4909 scope.go:122] "RemoveContainer" containerID="b118def65225af7f9db3a23cf610c0f055b6ea0a0891c63a193d38b393d2a62b"
Nov 03 13:00:03.462736 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:03.462677    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" podStartSLOduration=3.462662675 podStartE2EDuration="3.462662675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:03.458327778 +0000 UTC m=+4152.470794439" watchObservedRunningTime="2026-11-03 13:00:03.462662675 +0000 UTC m=+4152.475129337"
Nov 03 13:00:04.276754 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.276728    4909 generic.go:356] "Generic (PLEG): container finished" podID="a5660732-c985-4aff-ae7f-a019ca46e893" containerID="8ff6f7bb76dda8c48f691da7939549ae306411f2a47f6197e700b56251004fdd" exitCode=60
Nov 03 13:00:04.277161 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.276808    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" event={"ID":"a5660732-c985-4aff-ae7f-a019ca46e893","Type":"ContainerDied","Data":"8ff6f7bb76dda8c48f691da7939549ae306411f2a47f6197e700b56251004fdd"}
Nov 03 13:00:04.277161 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.276880    4909 scope.go:122] "RemoveContainer" containerID="9d4dc9c28cd78bba80afe2f97579f086930a13b4df1d3462134d5affe79bbb6b"
Nov 03 13:00:04.277161 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.277040    4909 scope.go:122] "RemoveContainer" containerID="8ff6f7bb76dda8c48f691da7939549ae306411f2a47f6197e700b56251004fdd"
Nov 03 13:00:04.277342 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:04.277323    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5_odh-ai-gateway-infra(a5660732-c985-4aff-ae7f-a019ca46e893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" podUID="a5660732-c985-4aff-ae7f-a019ca46e893"
Nov 03 13:00:04.279009 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.278986    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" event={"ID":"5fab2913-3757-4544-b72e-8219fe9dac4d","Type":"ContainerStarted","Data":"eab69569e1e46a6b8a5f9344b393da2627d3a1830158e6209fc2d5701271fd01"}
Nov 03 13:00:04.281011 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.280989    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" event={"ID":"f243926b-8892-434a-afed-9e62b602c3ca","Type":"ContainerStarted","Data":"3605dd6d6cc2ae606080d7315a7c3b3c43f3e03e7eecb681236ae3db94039e62"}
Nov 03 13:00:04.282932 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.282910    4909 generic.go:356] "Generic (PLEG): container finished" podID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerID="9b2466124798367661b1066efc02443f5f8f78e416f3cc94dd2a4e90fac23076" exitCode=60
Nov 03 13:00:04.283026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.282998    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" event={"ID":"44a5c80b-3ed7-4009-8c26-bc262fb07651","Type":"ContainerDied","Data":"9b2466124798367661b1066efc02443f5f8f78e416f3cc94dd2a4e90fac23076"}
Nov 03 13:00:04.283193 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.283177    4909 scope.go:122] "RemoveContainer" containerID="9b2466124798367661b1066efc02443f5f8f78e416f3cc94dd2a4e90fac23076"
Nov 03 13:00:04.283405 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:04.283387    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44_odh-ai-gateway-infra(44a5c80b-3ed7-4009-8c26-bc262fb07651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651"
Nov 03 13:00:04.285160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.285140    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" event={"ID":"01030a13-b2a2-423b-aa1e-bf92c969edbd","Type":"ContainerStarted","Data":"61b44e4352c4a46da5f53ffb58fe18adf8348cec41d4e916f0010573e78d7c25"}
Nov 03 13:00:04.286890 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.286870    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" event={"ID":"4b8e61c0-6280-462c-9d6b-95665790b662","Type":"ContainerStarted","Data":"536b9c66bd3fe8d9dbcd75f597d0a72db76ed5e06bac289a8de69b2dfe7d038a"}
Nov 03 13:00:04.288901 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.288879    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" event={"ID":"79bbbd2b-c562-4575-9e20-6bf81c2ce99e","Type":"ContainerStarted","Data":"38a314a68068cbf18b16339efcda92b7b7b9c549757eb974a2b8a311a9d4f28a"}
Nov 03 13:00:04.290758 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.290739    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" event={"ID":"9602ddec-e761-4948-95f0-96427993446d","Type":"ContainerStarted","Data":"73abe2c0862a004a3ed148add03a551a9f50beeb15a8d25dc7149896b38136cf"}
Nov 03 13:00:04.290972 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:04.290955    4909 scope.go:122] "RemoveContainer" containerID="59318f543907b0a317cd30554a811ba23daad7c7124beed57c9f65d084e2ffc5"
Nov 03 13:00:05.299263 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.299231    4909 generic.go:356] "Generic (PLEG): container finished" podID="9602ddec-e761-4948-95f0-96427993446d" containerID="73abe2c0862a004a3ed148add03a551a9f50beeb15a8d25dc7149896b38136cf" exitCode=60
Nov 03 13:00:05.299773 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.299312    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" event={"ID":"9602ddec-e761-4948-95f0-96427993446d","Type":"ContainerDied","Data":"73abe2c0862a004a3ed148add03a551a9f50beeb15a8d25dc7149896b38136cf"}
Nov 03 13:00:05.299773 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.299381    4909 scope.go:122] "RemoveContainer" containerID="381ec8939dd07679fe4b170940d476dcf321a06acd9d12b4f10980e4cfe98e44"
Nov 03 13:00:05.299773 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.299564    4909 scope.go:122] "RemoveContainer" containerID="73abe2c0862a004a3ed148add03a551a9f50beeb15a8d25dc7149896b38136cf"
Nov 03 13:00:05.299950 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.299920    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd_odh-ai-gateway-infra(9602ddec-e761-4948-95f0-96427993446d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" podUID="9602ddec-e761-4948-95f0-96427993446d"
Nov 03 13:00:05.301834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.301811    4909 scope.go:122] "RemoveContainer" containerID="8ff6f7bb76dda8c48f691da7939549ae306411f2a47f6197e700b56251004fdd"
Nov 03 13:00:05.302136 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.302112    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5_odh-ai-gateway-infra(a5660732-c985-4aff-ae7f-a019ca46e893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" podUID="a5660732-c985-4aff-ae7f-a019ca46e893"
Nov 03 13:00:05.303556 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.303526    4909 generic.go:356] "Generic (PLEG): container finished" podID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerID="eab69569e1e46a6b8a5f9344b393da2627d3a1830158e6209fc2d5701271fd01" exitCode=60
Nov 03 13:00:05.303656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.303617    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" event={"ID":"5fab2913-3757-4544-b72e-8219fe9dac4d","Type":"ContainerDied","Data":"eab69569e1e46a6b8a5f9344b393da2627d3a1830158e6209fc2d5701271fd01"}
Nov 03 13:00:05.303820 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.303799    4909 scope.go:122] "RemoveContainer" containerID="eab69569e1e46a6b8a5f9344b393da2627d3a1830158e6209fc2d5701271fd01"
Nov 03 13:00:05.304091 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.304067    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49_odh-ai-gateway-infra(5fab2913-3757-4544-b72e-8219fe9dac4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d"
Nov 03 13:00:05.305959 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.305941    4909 generic.go:356] "Generic (PLEG): container finished" podID="f243926b-8892-434a-afed-9e62b602c3ca" containerID="3605dd6d6cc2ae606080d7315a7c3b3c43f3e03e7eecb681236ae3db94039e62" exitCode=60
Nov 03 13:00:05.306259 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.306245    4909 scope.go:122] "RemoveContainer" containerID="3605dd6d6cc2ae606080d7315a7c3b3c43f3e03e7eecb681236ae3db94039e62"
Nov 03 13:00:05.306463 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.306446    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg_odh-ai-gateway-infra(f243926b-8892-434a-afed-9e62b602c3ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" podUID="f243926b-8892-434a-afed-9e62b602c3ca"
Nov 03 13:00:05.306505 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.306485    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" event={"ID":"f243926b-8892-434a-afed-9e62b602c3ca","Type":"ContainerDied","Data":"3605dd6d6cc2ae606080d7315a7c3b3c43f3e03e7eecb681236ae3db94039e62"}
Nov 03 13:00:05.308499 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.308481    4909 scope.go:122] "RemoveContainer" containerID="9b2466124798367661b1066efc02443f5f8f78e416f3cc94dd2a4e90fac23076"
Nov 03 13:00:05.308727 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.308676    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44_odh-ai-gateway-infra(44a5c80b-3ed7-4009-8c26-bc262fb07651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651"
Nov 03 13:00:05.310198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.310179    4909 generic.go:356] "Generic (PLEG): container finished" podID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerID="61b44e4352c4a46da5f53ffb58fe18adf8348cec41d4e916f0010573e78d7c25" exitCode=60
Nov 03 13:00:05.310304 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.310249    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" event={"ID":"01030a13-b2a2-423b-aa1e-bf92c969edbd","Type":"ContainerDied","Data":"61b44e4352c4a46da5f53ffb58fe18adf8348cec41d4e916f0010573e78d7c25"}
Nov 03 13:00:05.310445 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.310427    4909 scope.go:122] "RemoveContainer" containerID="61b44e4352c4a46da5f53ffb58fe18adf8348cec41d4e916f0010573e78d7c25"
Nov 03 13:00:05.310729 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.310693    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d_odh-ai-gateway-infra(01030a13-b2a2-423b-aa1e-bf92c969edbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd"
Nov 03 13:00:05.312160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.312137    4909 generic.go:356] "Generic (PLEG): container finished" podID="4b8e61c0-6280-462c-9d6b-95665790b662" containerID="536b9c66bd3fe8d9dbcd75f597d0a72db76ed5e06bac289a8de69b2dfe7d038a" exitCode=60
Nov 03 13:00:05.312273 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.312206    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" event={"ID":"4b8e61c0-6280-462c-9d6b-95665790b662","Type":"ContainerDied","Data":"536b9c66bd3fe8d9dbcd75f597d0a72db76ed5e06bac289a8de69b2dfe7d038a"}
Nov 03 13:00:05.312388 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.312373    4909 scope.go:122] "RemoveContainer" containerID="536b9c66bd3fe8d9dbcd75f597d0a72db76ed5e06bac289a8de69b2dfe7d038a"
Nov 03 13:00:05.312590 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.312569    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr_odh-ai-gateway-infra(4b8e61c0-6280-462c-9d6b-95665790b662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" podUID="4b8e61c0-6280-462c-9d6b-95665790b662"
Nov 03 13:00:05.314057 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.314036    4909 generic.go:356] "Generic (PLEG): container finished" podID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerID="38a314a68068cbf18b16339efcda92b7b7b9c549757eb974a2b8a311a9d4f28a" exitCode=60
Nov 03 13:00:05.314140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.314078    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" event={"ID":"79bbbd2b-c562-4575-9e20-6bf81c2ce99e","Type":"ContainerDied","Data":"38a314a68068cbf18b16339efcda92b7b7b9c549757eb974a2b8a311a9d4f28a"}
Nov 03 13:00:05.314274 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.314262    4909 scope.go:122] "RemoveContainer" containerID="38a314a68068cbf18b16339efcda92b7b7b9c549757eb974a2b8a311a9d4f28a"
Nov 03 13:00:05.314454 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:05.314439    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j_odh-ai-gateway-infra(79bbbd2b-c562-4575-9e20-6bf81c2ce99e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e"
Nov 03 13:00:05.317744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.317723    4909 scope.go:122] "RemoveContainer" containerID="dd7962686955368736c00373afe0f033b547c418590cb44b93f4cad9f53621a3"
Nov 03 13:00:05.330928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.330909    4909 scope.go:122] "RemoveContainer" containerID="b118def65225af7f9db3a23cf610c0f055b6ea0a0891c63a193d38b393d2a62b"
Nov 03 13:00:05.344776 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.344759    4909 scope.go:122] "RemoveContainer" containerID="40f47979f6fffbe5d0d3ddcec72e2705a3567afc0d9bc6b2ef948dfd3b2c1596"
Nov 03 13:00:05.357765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.357746    4909 scope.go:122] "RemoveContainer" containerID="93ea2ebbdbaecf1386daacfd62559d0d34eae7a5292cbce26a3ae9bad786007e"
Nov 03 13:00:05.370053 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.370037    4909 scope.go:122] "RemoveContainer" containerID="733c4c833bf85f2f0b105b48a095f26f2730d9d7e125194d54d0af8a1d482804"
Nov 03 13:00:05.771210 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:05.771171    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l" podUID="36097829-e13a-4805-9a9e-e020c0b2fa4c" containerName="payload-processing" containerID="cri-o://a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84" gracePeriod=25
Nov 03 13:00:06.062088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.062060    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 13:00:06.137639 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.137610    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36097829-e13a-4805-9a9e-e020c0b2fa4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36097829-e13a-4805-9a9e-e020c0b2fa4c-plugins-config-volume\") pod \"36097829-e13a-4805-9a9e-e020c0b2fa4c\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") "
Nov 03 13:00:06.137794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.137773    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36097829-e13a-4805-9a9e-e020c0b2fa4c-kube-api-access-l8557\" (UniqueName: \"kubernetes.io/projected/36097829-e13a-4805-9a9e-e020c0b2fa4c-kube-api-access-l8557\") pod \"36097829-e13a-4805-9a9e-e020c0b2fa4c\" (UID: \"36097829-e13a-4805-9a9e-e020c0b2fa4c\") "
Nov 03 13:00:06.138015 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.137994    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36097829-e13a-4805-9a9e-e020c0b2fa4c-plugins-config-volume" pod "36097829-e13a-4805-9a9e-e020c0b2fa4c" (UID: "36097829-e13a-4805-9a9e-e020c0b2fa4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:06.138202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.138183    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36097829-e13a-4805-9a9e-e020c0b2fa4c-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:06.139829 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.139807    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36097829-e13a-4805-9a9e-e020c0b2fa4c-kube-api-access-l8557" pod "36097829-e13a-4805-9a9e-e020c0b2fa4c" (UID: "36097829-e13a-4805-9a9e-e020c0b2fa4c"). InnerVolumeSpecName "kube-api-access-l8557". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:06.239785 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.239754    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8557\" (UniqueName: \"kubernetes.io/projected/36097829-e13a-4805-9a9e-e020c0b2fa4c-kube-api-access-l8557\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:06.323012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.322936    4909 scope.go:122] "RemoveContainer" containerID="61b44e4352c4a46da5f53ffb58fe18adf8348cec41d4e916f0010573e78d7c25"
Nov 03 13:00:06.323435 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:06.323272    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d_odh-ai-gateway-infra(01030a13-b2a2-423b-aa1e-bf92c969edbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd"
Nov 03 13:00:06.324512 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.324490    4909 scope.go:122] "RemoveContainer" containerID="536b9c66bd3fe8d9dbcd75f597d0a72db76ed5e06bac289a8de69b2dfe7d038a"
Nov 03 13:00:06.324820 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:06.324794    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr_odh-ai-gateway-infra(4b8e61c0-6280-462c-9d6b-95665790b662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" podUID="4b8e61c0-6280-462c-9d6b-95665790b662"
Nov 03 13:00:06.326085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.326047    4909 scope.go:122] "RemoveContainer" containerID="38a314a68068cbf18b16339efcda92b7b7b9c549757eb974a2b8a311a9d4f28a"
Nov 03 13:00:06.326367 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:06.326347    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j_odh-ai-gateway-infra(79bbbd2b-c562-4575-9e20-6bf81c2ce99e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e"
Nov 03 13:00:06.327601 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.327574    4909 scope.go:122] "RemoveContainer" containerID="73abe2c0862a004a3ed148add03a551a9f50beeb15a8d25dc7149896b38136cf"
Nov 03 13:00:06.327931 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:06.327907    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd_odh-ai-gateway-infra(9602ddec-e761-4948-95f0-96427993446d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" podUID="9602ddec-e761-4948-95f0-96427993446d"
Nov 03 13:00:06.329017 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.329001    4909 scope.go:122] "RemoveContainer" containerID="eab69569e1e46a6b8a5f9344b393da2627d3a1830158e6209fc2d5701271fd01"
Nov 03 13:00:06.329257 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:06.329234    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49_odh-ai-gateway-infra(5fab2913-3757-4544-b72e-8219fe9dac4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d"
Nov 03 13:00:06.330381 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.330366    4909 scope.go:122] "RemoveContainer" containerID="3605dd6d6cc2ae606080d7315a7c3b3c43f3e03e7eecb681236ae3db94039e62"
Nov 03 13:00:06.330598 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:06.330581    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg_odh-ai-gateway-infra(f243926b-8892-434a-afed-9e62b602c3ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" podUID="f243926b-8892-434a-afed-9e62b602c3ca"
Nov 03 13:00:06.331426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.331408    4909 generic.go:356] "Generic (PLEG): container finished" podID="36097829-e13a-4805-9a9e-e020c0b2fa4c" containerID="a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84" exitCode=0
Nov 03 13:00:06.331515 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.331449    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"
Nov 03 13:00:06.331515 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.331484    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l" event={"ID":"36097829-e13a-4805-9a9e-e020c0b2fa4c","Type":"ContainerDied","Data":"a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84"}
Nov 03 13:00:06.331587 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.331528    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l" event={"ID":"36097829-e13a-4805-9a9e-e020c0b2fa4c","Type":"ContainerDied","Data":"5eae79e18f03adb99b4d27c60a331d074efec0a6758243533f48e9ef6bde7db7"}
Nov 03 13:00:06.331587 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.331552    4909 scope.go:122] "RemoveContainer" containerID="a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84"
Nov 03 13:00:06.345433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.345412    4909 scope.go:122] "RemoveContainer" containerID="a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84"
Nov 03 13:00:06.345768 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:06.345688    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84\": container with ID starting with a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84 not found: ID does not exist" containerID="a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84"
Nov 03 13:00:06.345846 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:06.345775    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84"} err="failed to get container status \"a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84\": rpc error: code = NotFound desc = could not find container \"a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84\": container with ID starting with a4293c11a38300b670d9947546ed3aeac8b4875c1eae6826e3440481d71fad84 not found: ID does not exist"
Nov 03 13:00:07.043812 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:07.043769    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"]
Nov 03 13:00:07.048584 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:07.048547    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-532442-57c9c7487c-84r9l"]
Nov 03 13:00:07.124976 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:07.124945    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36097829-e13a-4805-9a9e-e020c0b2fa4c" path="/var/lib/kubelet/pods/36097829-e13a-4805-9a9e-e020c0b2fa4c/volumes"
Nov 03 13:00:09.565435 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:09.565399    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:00:10.941836 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:10.941806    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"]
Nov 03 13:00:15.942990 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:15.942939    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc" podUID="570ef14d-3b08-46d6-840d-7281aa3b6e21" containerName="payload-processing" containerID="cri-o://aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b" gracePeriod=25
Nov 03 13:00:16.116710 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.116659    4909 scope.go:122] "RemoveContainer" containerID="61b44e4352c4a46da5f53ffb58fe18adf8348cec41d4e916f0010573e78d7c25"
Nov 03 13:00:16.116985 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.116963    4909 scope.go:122] "RemoveContainer" containerID="eab69569e1e46a6b8a5f9344b393da2627d3a1830158e6209fc2d5701271fd01"
Nov 03 13:00:16.117072 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.117026    4909 scope.go:122] "RemoveContainer" containerID="9b2466124798367661b1066efc02443f5f8f78e416f3cc94dd2a4e90fac23076"
Nov 03 13:00:16.248079 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.248055    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 13:00:16.277656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.277629    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570ef14d-3b08-46d6-840d-7281aa3b6e21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570ef14d-3b08-46d6-840d-7281aa3b6e21-plugins-config-volume\") pod \"570ef14d-3b08-46d6-840d-7281aa3b6e21\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") "
Nov 03 13:00:16.277870 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.277852    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570ef14d-3b08-46d6-840d-7281aa3b6e21-kube-api-access-7lsv4\" (UniqueName: \"kubernetes.io/projected/570ef14d-3b08-46d6-840d-7281aa3b6e21-kube-api-access-7lsv4\") pod \"570ef14d-3b08-46d6-840d-7281aa3b6e21\" (UID: \"570ef14d-3b08-46d6-840d-7281aa3b6e21\") "
Nov 03 13:00:16.277985 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.277964    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570ef14d-3b08-46d6-840d-7281aa3b6e21-plugins-config-volume" pod "570ef14d-3b08-46d6-840d-7281aa3b6e21" (UID: "570ef14d-3b08-46d6-840d-7281aa3b6e21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:16.278301 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.278285    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570ef14d-3b08-46d6-840d-7281aa3b6e21-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:16.340624 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.340595    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570ef14d-3b08-46d6-840d-7281aa3b6e21-kube-api-access-7lsv4" pod "570ef14d-3b08-46d6-840d-7281aa3b6e21" (UID: "570ef14d-3b08-46d6-840d-7281aa3b6e21"). InnerVolumeSpecName "kube-api-access-7lsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:16.380444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.380414    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lsv4\" (UniqueName: \"kubernetes.io/projected/570ef14d-3b08-46d6-840d-7281aa3b6e21-kube-api-access-7lsv4\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:16.462935 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.462876    4909 generic.go:356] "Generic (PLEG): container finished" podID="570ef14d-3b08-46d6-840d-7281aa3b6e21" containerID="aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b" exitCode=0
Nov 03 13:00:16.462935 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.462918    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"
Nov 03 13:00:16.463082 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.462950    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc" event={"ID":"570ef14d-3b08-46d6-840d-7281aa3b6e21","Type":"ContainerDied","Data":"aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b"}
Nov 03 13:00:16.463082 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.463002    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc" event={"ID":"570ef14d-3b08-46d6-840d-7281aa3b6e21","Type":"ContainerDied","Data":"af9374e6d2c524eb4df8edc686f4b0df70226e1108c7f5ffa886c56b92c4f602"}
Nov 03 13:00:16.463082 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.463024    4909 scope.go:122] "RemoveContainer" containerID="aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b"
Nov 03 13:00:16.473731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.473712    4909 scope.go:122] "RemoveContainer" containerID="aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b"
Nov 03 13:00:16.473994 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:16.473977    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b\": container with ID starting with aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b not found: ID does not exist" containerID="aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b"
Nov 03 13:00:16.474076 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.474001    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b"} err="failed to get container status \"aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b\": rpc error: code = NotFound desc = could not find container \"aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b\": container with ID starting with aeba5abb4595a809ea295ca758f60a5794248229c05b77d05f27432ef187ce9b not found: ID does not exist"
Nov 03 13:00:16.616938 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.616908    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"]
Nov 03 13:00:16.622502 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:16.622475    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b737fe-67466b9c75-6z9cc"]
Nov 03 13:00:17.116420 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.116390    4909 scope.go:122] "RemoveContainer" containerID="38a314a68068cbf18b16339efcda92b7b7b9c549757eb974a2b8a311a9d4f28a"
Nov 03 13:00:17.116930 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.116497    4909 scope.go:122] "RemoveContainer" containerID="73abe2c0862a004a3ed148add03a551a9f50beeb15a8d25dc7149896b38136cf"
Nov 03 13:00:17.126782 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.126758    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570ef14d-3b08-46d6-840d-7281aa3b6e21" path="/var/lib/kubelet/pods/570ef14d-3b08-46d6-840d-7281aa3b6e21/volumes"
Nov 03 13:00:17.471498 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.471471    4909 generic.go:356] "Generic (PLEG): container finished" podID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerID="ee716fd2fc51a6680b256fd0e94d1725b356c6348c32dede426526af913252f8" exitCode=60
Nov 03 13:00:17.471629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.471551    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" event={"ID":"44a5c80b-3ed7-4009-8c26-bc262fb07651","Type":"ContainerDied","Data":"ee716fd2fc51a6680b256fd0e94d1725b356c6348c32dede426526af913252f8"}
Nov 03 13:00:17.471629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.471616    4909 scope.go:122] "RemoveContainer" containerID="9b2466124798367661b1066efc02443f5f8f78e416f3cc94dd2a4e90fac23076"
Nov 03 13:00:17.472129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.472106    4909 scope.go:122] "RemoveContainer" containerID="ee716fd2fc51a6680b256fd0e94d1725b356c6348c32dede426526af913252f8"
Nov 03 13:00:17.472503 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:17.472477    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44_odh-ai-gateway-infra(44a5c80b-3ed7-4009-8c26-bc262fb07651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651"
Nov 03 13:00:17.474016 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.473994    4909 generic.go:356] "Generic (PLEG): container finished" podID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerID="a9c8e0b38a3ad6ab2ad38b00b2ccc0b1ef650a6d6c2c8fb1960dba9f01ffa341" exitCode=60
Nov 03 13:00:17.474149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.474071    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" event={"ID":"01030a13-b2a2-423b-aa1e-bf92c969edbd","Type":"ContainerDied","Data":"a9c8e0b38a3ad6ab2ad38b00b2ccc0b1ef650a6d6c2c8fb1960dba9f01ffa341"}
Nov 03 13:00:17.474473 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.474449    4909 scope.go:122] "RemoveContainer" containerID="a9c8e0b38a3ad6ab2ad38b00b2ccc0b1ef650a6d6c2c8fb1960dba9f01ffa341"
Nov 03 13:00:17.474862 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:17.474832    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d_odh-ai-gateway-infra(01030a13-b2a2-423b-aa1e-bf92c969edbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd"
Nov 03 13:00:17.477194 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.477176    4909 generic.go:356] "Generic (PLEG): container finished" podID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerID="99d3ff4c808293914235491262fa8d0b2689e4d2c98e5dd7104a7001395d9c4f" exitCode=60
Nov 03 13:00:17.477304 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.477207    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" event={"ID":"5fab2913-3757-4544-b72e-8219fe9dac4d","Type":"ContainerDied","Data":"99d3ff4c808293914235491262fa8d0b2689e4d2c98e5dd7104a7001395d9c4f"}
Nov 03 13:00:17.477578 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.477555    4909 scope.go:122] "RemoveContainer" containerID="99d3ff4c808293914235491262fa8d0b2689e4d2c98e5dd7104a7001395d9c4f"
Nov 03 13:00:17.477851 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:17.477831    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49_odh-ai-gateway-infra(5fab2913-3757-4544-b72e-8219fe9dac4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d"
Nov 03 13:00:17.502791 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.502774    4909 scope.go:122] "RemoveContainer" containerID="61b44e4352c4a46da5f53ffb58fe18adf8348cec41d4e916f0010573e78d7c25"
Nov 03 13:00:17.515850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:17.515827    4909 scope.go:122] "RemoveContainer" containerID="eab69569e1e46a6b8a5f9344b393da2627d3a1830158e6209fc2d5701271fd01"
Nov 03 13:00:18.116829 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.116802    4909 scope.go:122] "RemoveContainer" containerID="536b9c66bd3fe8d9dbcd75f597d0a72db76ed5e06bac289a8de69b2dfe7d038a"
Nov 03 13:00:18.117322 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.116875    4909 scope.go:122] "RemoveContainer" containerID="8ff6f7bb76dda8c48f691da7939549ae306411f2a47f6197e700b56251004fdd"
Nov 03 13:00:18.117322 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.117156    4909 scope.go:122] "RemoveContainer" containerID="3605dd6d6cc2ae606080d7315a7c3b3c43f3e03e7eecb681236ae3db94039e62"
Nov 03 13:00:18.490039 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.490012    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" event={"ID":"a5660732-c985-4aff-ae7f-a019ca46e893","Type":"ContainerStarted","Data":"5ef04c0ca72fac70130fc1582fc4eeb30077e1db52829593150cad889b560f42"}
Nov 03 13:00:18.496513 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.496490    4909 generic.go:356] "Generic (PLEG): container finished" podID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerID="f82eebe55713be8c86184d3d42904b7632e2a02261201038e01717e81e891fe9" exitCode=60
Nov 03 13:00:18.496636 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.496556    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" event={"ID":"79bbbd2b-c562-4575-9e20-6bf81c2ce99e","Type":"ContainerDied","Data":"f82eebe55713be8c86184d3d42904b7632e2a02261201038e01717e81e891fe9"}
Nov 03 13:00:18.496636 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.496600    4909 scope.go:122] "RemoveContainer" containerID="38a314a68068cbf18b16339efcda92b7b7b9c549757eb974a2b8a311a9d4f28a"
Nov 03 13:00:18.496951 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.496935    4909 scope.go:122] "RemoveContainer" containerID="f82eebe55713be8c86184d3d42904b7632e2a02261201038e01717e81e891fe9"
Nov 03 13:00:18.497246 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:18.497227    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j_odh-ai-gateway-infra(79bbbd2b-c562-4575-9e20-6bf81c2ce99e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e"
Nov 03 13:00:18.498541 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.498521    4909 generic.go:356] "Generic (PLEG): container finished" podID="9602ddec-e761-4948-95f0-96427993446d" containerID="d4ed592f9614a64d9452b32ec215910bf82c7440924d2365889f00ba95f5d668" exitCode=60
Nov 03 13:00:18.498657 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.498550    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" event={"ID":"9602ddec-e761-4948-95f0-96427993446d","Type":"ContainerDied","Data":"d4ed592f9614a64d9452b32ec215910bf82c7440924d2365889f00ba95f5d668"}
Nov 03 13:00:18.498945 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.498928    4909 scope.go:122] "RemoveContainer" containerID="d4ed592f9614a64d9452b32ec215910bf82c7440924d2365889f00ba95f5d668"
Nov 03 13:00:18.499213 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:18.499192    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd_odh-ai-gateway-infra(9602ddec-e761-4948-95f0-96427993446d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" podUID="9602ddec-e761-4948-95f0-96427993446d"
Nov 03 13:00:18.509053 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.509032    4909 scope.go:122] "RemoveContainer" containerID="73abe2c0862a004a3ed148add03a551a9f50beeb15a8d25dc7149896b38136cf"
Nov 03 13:00:18.587112 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.587087    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"]
Nov 03 13:00:18.697310 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.697278    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"]
Nov 03 13:00:18.765895 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.765869    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:18.796184 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.796135    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"]
Nov 03 13:00:18.819739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.819710    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a5c80b-3ed7-4009-8c26-bc262fb07651-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a5c80b-3ed7-4009-8c26-bc262fb07651-maas-api-service-ca\") pod \"44a5c80b-3ed7-4009-8c26-bc262fb07651\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") "
Nov 03 13:00:18.820191 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.819931    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a5c80b-3ed7-4009-8c26-bc262fb07651-kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/44a5c80b-3ed7-4009-8c26-bc262fb07651-kube-api-access-8qxsx\") pod \"44a5c80b-3ed7-4009-8c26-bc262fb07651\" (UID: \"44a5c80b-3ed7-4009-8c26-bc262fb07651\") "
Nov 03 13:00:18.820610 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.820570    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a5c80b-3ed7-4009-8c26-bc262fb07651-maas-api-service-ca" pod "44a5c80b-3ed7-4009-8c26-bc262fb07651" (UID: "44a5c80b-3ed7-4009-8c26-bc262fb07651"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:18.838359 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.838293    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a5c80b-3ed7-4009-8c26-bc262fb07651-kube-api-access-8qxsx" pod "44a5c80b-3ed7-4009-8c26-bc262fb07651" (UID: "44a5c80b-3ed7-4009-8c26-bc262fb07651"). InnerVolumeSpecName "kube-api-access-8qxsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:18.921985 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.921914    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a5c80b-3ed7-4009-8c26-bc262fb07651-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:18.921985 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.921942    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/44a5c80b-3ed7-4009-8c26-bc262fb07651-kube-api-access-8qxsx\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:18.946178 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.946150    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:18.967619 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:18.967592    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:19.023157 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.023129    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01030a13-b2a2-423b-aa1e-bf92c969edbd-kube-api-access-pmn9p\" (UniqueName: \"kubernetes.io/projected/01030a13-b2a2-423b-aa1e-bf92c969edbd-kube-api-access-pmn9p\") pod \"01030a13-b2a2-423b-aa1e-bf92c969edbd\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") "
Nov 03 13:00:19.023298 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.023169    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fab2913-3757-4544-b72e-8219fe9dac4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fab2913-3757-4544-b72e-8219fe9dac4d-maas-api-service-ca\") pod \"5fab2913-3757-4544-b72e-8219fe9dac4d\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") "
Nov 03 13:00:19.023298 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.023217    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fab2913-3757-4544-b72e-8219fe9dac4d-kube-api-access-8tv66\" (UniqueName: \"kubernetes.io/projected/5fab2913-3757-4544-b72e-8219fe9dac4d-kube-api-access-8tv66\") pod \"5fab2913-3757-4544-b72e-8219fe9dac4d\" (UID: \"5fab2913-3757-4544-b72e-8219fe9dac4d\") "
Nov 03 13:00:19.023404 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.023295    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01030a13-b2a2-423b-aa1e-bf92c969edbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01030a13-b2a2-423b-aa1e-bf92c969edbd-maas-api-service-ca\") pod \"01030a13-b2a2-423b-aa1e-bf92c969edbd\" (UID: \"01030a13-b2a2-423b-aa1e-bf92c969edbd\") "
Nov 03 13:00:19.023667 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.023635    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fab2913-3757-4544-b72e-8219fe9dac4d-maas-api-service-ca" pod "5fab2913-3757-4544-b72e-8219fe9dac4d" (UID: "5fab2913-3757-4544-b72e-8219fe9dac4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:19.023802 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.023670    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01030a13-b2a2-423b-aa1e-bf92c969edbd-maas-api-service-ca" pod "01030a13-b2a2-423b-aa1e-bf92c969edbd" (UID: "01030a13-b2a2-423b-aa1e-bf92c969edbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:19.024092 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.024071    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01030a13-b2a2-423b-aa1e-bf92c969edbd-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.024139 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.024098    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fab2913-3757-4544-b72e-8219fe9dac4d-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.025410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.025386    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01030a13-b2a2-423b-aa1e-bf92c969edbd-kube-api-access-pmn9p" pod "01030a13-b2a2-423b-aa1e-bf92c969edbd" (UID: "01030a13-b2a2-423b-aa1e-bf92c969edbd"). InnerVolumeSpecName "kube-api-access-pmn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:19.025754 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.025738    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fab2913-3757-4544-b72e-8219fe9dac4d-kube-api-access-8tv66" pod "5fab2913-3757-4544-b72e-8219fe9dac4d" (UID: "5fab2913-3757-4544-b72e-8219fe9dac4d"). InnerVolumeSpecName "kube-api-access-8tv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:19.124964 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.124938    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmn9p\" (UniqueName: \"kubernetes.io/projected/01030a13-b2a2-423b-aa1e-bf92c969edbd-kube-api-access-pmn9p\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.124964 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.124962    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tv66\" (UniqueName: \"kubernetes.io/projected/5fab2913-3757-4544-b72e-8219fe9dac4d-kube-api-access-8tv66\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.515338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.515242    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d" event={"ID":"01030a13-b2a2-423b-aa1e-bf92c969edbd","Type":"ContainerDied","Data":"3742b7d580bbb085827302fcbd86d0f5d62a0e273e9b458dfc6c3fcd8b8a52e2"}
Nov 03 13:00:19.515532 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.515358    4909 scope.go:122] "RemoveContainer" containerID="a9c8e0b38a3ad6ab2ad38b00b2ccc0b1ef650a6d6c2c8fb1960dba9f01ffa341"
Nov 03 13:00:19.515652 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.515264    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"
Nov 03 13:00:19.518357 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.518332    4909 generic.go:356] "Generic (PLEG): container finished" podID="4b8e61c0-6280-462c-9d6b-95665790b662" containerID="0b40d31c70f496a48a1d123e957891ab56d39eda64381dce361872b06f26df21" exitCode=60
Nov 03 13:00:19.518528 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.518452    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" event={"ID":"4b8e61c0-6280-462c-9d6b-95665790b662","Type":"ContainerDied","Data":"0b40d31c70f496a48a1d123e957891ab56d39eda64381dce361872b06f26df21"}
Nov 03 13:00:19.518978 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.518928    4909 scope.go:122] "RemoveContainer" containerID="0b40d31c70f496a48a1d123e957891ab56d39eda64381dce361872b06f26df21"
Nov 03 13:00:19.519261 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:19.519224    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr_odh-ai-gateway-infra(4b8e61c0-6280-462c-9d6b-95665790b662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" podUID="4b8e61c0-6280-462c-9d6b-95665790b662"
Nov 03 13:00:19.525264 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.525227    4909 generic.go:356] "Generic (PLEG): container finished" podID="a5660732-c985-4aff-ae7f-a019ca46e893" containerID="5ef04c0ca72fac70130fc1582fc4eeb30077e1db52829593150cad889b560f42" exitCode=60
Nov 03 13:00:19.525385 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.525322    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" event={"ID":"a5660732-c985-4aff-ae7f-a019ca46e893","Type":"ContainerDied","Data":"5ef04c0ca72fac70130fc1582fc4eeb30077e1db52829593150cad889b560f42"}
Nov 03 13:00:19.525667 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.525649    4909 scope.go:122] "RemoveContainer" containerID="5ef04c0ca72fac70130fc1582fc4eeb30077e1db52829593150cad889b560f42"
Nov 03 13:00:19.526009 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:19.525984    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5_odh-ai-gateway-infra(a5660732-c985-4aff-ae7f-a019ca46e893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" podUID="a5660732-c985-4aff-ae7f-a019ca46e893"
Nov 03 13:00:19.527897 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.527863    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49" event={"ID":"5fab2913-3757-4544-b72e-8219fe9dac4d","Type":"ContainerDied","Data":"870da8da4a596a3a1be3eb0d67ba757c28413685a9168d4f25f98e2333e97b68"}
Nov 03 13:00:19.528105 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.528090    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"
Nov 03 13:00:19.532106 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.532060    4909 scope.go:122] "RemoveContainer" containerID="536b9c66bd3fe8d9dbcd75f597d0a72db76ed5e06bac289a8de69b2dfe7d038a"
Nov 03 13:00:19.533230 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.533208    4909 generic.go:356] "Generic (PLEG): container finished" podID="f243926b-8892-434a-afed-9e62b602c3ca" containerID="48460781df8939890bf229c70ad95b1f4a17c44711458f1d7d718578f63ca89f" exitCode=60
Nov 03 13:00:19.533585 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.533562    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" event={"ID":"f243926b-8892-434a-afed-9e62b602c3ca","Type":"ContainerDied","Data":"48460781df8939890bf229c70ad95b1f4a17c44711458f1d7d718578f63ca89f"}
Nov 03 13:00:19.534175 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.534159    4909 scope.go:122] "RemoveContainer" containerID="48460781df8939890bf229c70ad95b1f4a17c44711458f1d7d718578f63ca89f"
Nov 03 13:00:19.534525 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:19.534500    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg_odh-ai-gateway-infra(f243926b-8892-434a-afed-9e62b602c3ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" podUID="f243926b-8892-434a-afed-9e62b602c3ca"
Nov 03 13:00:19.535446 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.535414    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44" event={"ID":"44a5c80b-3ed7-4009-8c26-bc262fb07651","Type":"ContainerDied","Data":"5d4c16836094f68a2c161205ecf2dfbf8adeb6a47a259f3c78bc086552daf502"}
Nov 03 13:00:19.535446 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.535434    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"
Nov 03 13:00:19.546243 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.546216    4909 scope.go:122] "RemoveContainer" containerID="8ff6f7bb76dda8c48f691da7939549ae306411f2a47f6197e700b56251004fdd"
Nov 03 13:00:19.559436 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.559418    4909 scope.go:122] "RemoveContainer" containerID="99d3ff4c808293914235491262fa8d0b2689e4d2c98e5dd7104a7001395d9c4f"
Nov 03 13:00:19.567978 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.567961    4909 scope.go:122] "RemoveContainer" containerID="3605dd6d6cc2ae606080d7315a7c3b3c43f3e03e7eecb681236ae3db94039e62"
Nov 03 13:00:19.580790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.580771    4909 scope.go:122] "RemoveContainer" containerID="ee716fd2fc51a6680b256fd0e94d1725b356c6348c32dede426526af913252f8"
Nov 03 13:00:19.610725 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.610671    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"]
Nov 03 13:00:19.717847 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.717813    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"]
Nov 03 13:00:19.908771 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.883797    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"]
Nov 03 13:00:19.988293 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.985856    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"]
Nov 03 13:00:19.988293 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:19.988165    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895180-cdb49"]
Nov 03 13:00:20.037325 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.037255    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:20.057980 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.057951    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"]
Nov 03 13:00:20.061670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.061642    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895180-bcm9d"]
Nov 03 13:00:20.081744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.081715    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-kube-api-access-qtqjv\" (UniqueName: \"kubernetes.io/projected/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-kube-api-access-qtqjv\") pod \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") "
Nov 03 13:00:20.081879 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.081842    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-maas-api-service-ca\") pod \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\" (UID: \"79bbbd2b-c562-4575-9e20-6bf81c2ce99e\") "
Nov 03 13:00:20.082237 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.082214    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-maas-api-service-ca" pod "79bbbd2b-c562-4575-9e20-6bf81c2ce99e" (UID: "79bbbd2b-c562-4575-9e20-6bf81c2ce99e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:20.083068 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.083043    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.085051 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.084329    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-kube-api-access-qtqjv" pod "79bbbd2b-c562-4575-9e20-6bf81c2ce99e" (UID: "79bbbd2b-c562-4575-9e20-6bf81c2ce99e"). InnerVolumeSpecName "kube-api-access-qtqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:20.160290 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.160232    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"]
Nov 03 13:00:20.167400 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.167373    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895180-j6j44"]
Nov 03 13:00:20.184215 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.184182    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtqjv\" (UniqueName: \"kubernetes.io/projected/79bbbd2b-c562-4575-9e20-6bf81c2ce99e-kube-api-access-qtqjv\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.196794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.196774    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:20.285448 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.285422    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9602ddec-e761-4948-95f0-96427993446d-kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/9602ddec-e761-4948-95f0-96427993446d-kube-api-access-ptccw\") pod \"9602ddec-e761-4948-95f0-96427993446d\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") "
Nov 03 13:00:20.285542 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.285458    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9602ddec-e761-4948-95f0-96427993446d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9602ddec-e761-4948-95f0-96427993446d-maas-api-service-ca\") pod \"9602ddec-e761-4948-95f0-96427993446d\" (UID: \"9602ddec-e761-4948-95f0-96427993446d\") "
Nov 03 13:00:20.285880 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.285849    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9602ddec-e761-4948-95f0-96427993446d-maas-api-service-ca" pod "9602ddec-e761-4948-95f0-96427993446d" (UID: "9602ddec-e761-4948-95f0-96427993446d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:20.287607 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.287566    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9602ddec-e761-4948-95f0-96427993446d-kube-api-access-ptccw" pod "9602ddec-e761-4948-95f0-96427993446d" (UID: "9602ddec-e761-4948-95f0-96427993446d"). InnerVolumeSpecName "kube-api-access-ptccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:20.396721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.395044    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/9602ddec-e761-4948-95f0-96427993446d-kube-api-access-ptccw\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.396721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.395085    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9602ddec-e761-4948-95f0-96427993446d-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.548383 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.548307    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"
Nov 03 13:00:20.548553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.548308    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j" event={"ID":"79bbbd2b-c562-4575-9e20-6bf81c2ce99e","Type":"ContainerDied","Data":"319318051d31cd76460739048cec59f8a0503e2c804e767ab277519b19d27670"}
Nov 03 13:00:20.548553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.548467    4909 scope.go:122] "RemoveContainer" containerID="f82eebe55713be8c86184d3d42904b7632e2a02261201038e01717e81e891fe9"
Nov 03 13:00:20.550488 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.550470    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"
Nov 03 13:00:20.550621 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.550514    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd" event={"ID":"9602ddec-e761-4948-95f0-96427993446d","Type":"ContainerDied","Data":"276a0ee26eec442e1bbb67b04396df0c37f77b64ee16a0c3b929effd9c0b5c29"}
Nov 03 13:00:20.566620 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.566594    4909 scope.go:122] "RemoveContainer" containerID="d4ed592f9614a64d9452b32ec215910bf82c7440924d2365889f00ba95f5d668"
Nov 03 13:00:20.711845 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.711820    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:20.714520 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.714502    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5660732-c985-4aff-ae7f-a019ca46e893-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5660732-c985-4aff-ae7f-a019ca46e893-maas-api-service-ca\") pod \"a5660732-c985-4aff-ae7f-a019ca46e893\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") "
Nov 03 13:00:20.714679 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.714666    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5660732-c985-4aff-ae7f-a019ca46e893-kube-api-access-tp8m8\" (UniqueName: \"kubernetes.io/projected/a5660732-c985-4aff-ae7f-a019ca46e893-kube-api-access-tp8m8\") pod \"a5660732-c985-4aff-ae7f-a019ca46e893\" (UID: \"a5660732-c985-4aff-ae7f-a019ca46e893\") "
Nov 03 13:00:20.714857 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.714837    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5660732-c985-4aff-ae7f-a019ca46e893-maas-api-service-ca" pod "a5660732-c985-4aff-ae7f-a019ca46e893" (UID: "a5660732-c985-4aff-ae7f-a019ca46e893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:20.715253 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.715217    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5660732-c985-4aff-ae7f-a019ca46e893-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.716902 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.716881    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5660732-c985-4aff-ae7f-a019ca46e893-kube-api-access-tp8m8" pod "a5660732-c985-4aff-ae7f-a019ca46e893" (UID: "a5660732-c985-4aff-ae7f-a019ca46e893"). InnerVolumeSpecName "kube-api-access-tp8m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:20.790136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.790111    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"]
Nov 03 13:00:20.808876 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.808791    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"]
Nov 03 13:00:20.815227 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.815201    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895180-cr79j"]
Nov 03 13:00:20.816068 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.816048    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp8m8\" (UniqueName: \"kubernetes.io/projected/a5660732-c985-4aff-ae7f-a019ca46e893-kube-api-access-tp8m8\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.915307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.915264    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"]
Nov 03 13:00:20.919519 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.919491    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895180-m66hd"]
Nov 03 13:00:20.925309 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.925288    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:20.954134 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954112    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"]
Nov 03 13:00:20.954780 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954766    4909 cpu_manager.go:414] "containerMap: removing container" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954782    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954796    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954802    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954810    4909 cpu_manager.go:414] "containerMap: removing container" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954815    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954823    4909 cpu_manager.go:414] "containerMap: removing container" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954828    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.954832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954835    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954840    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954851    4909 cpu_manager.go:414] "containerMap: removing container" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954855    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954861    4909 cpu_manager.go:414] "containerMap: removing container" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954865    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954873    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954877    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954885    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954889    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954899    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954903    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954911    4909 cpu_manager.go:414] "containerMap: removing container" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954916    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954922    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954926    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954932    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954936    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954943    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954947    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954956    4909 cpu_manager.go:414] "containerMap: removing container" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954960    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954968    4909 cpu_manager.go:414] "containerMap: removing container" podUID="36097829-e13a-4805-9a9e-e020c0b2fa4c" containerName="payload-processing"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954973    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36097829-e13a-4805-9a9e-e020c0b2fa4c" containerName="payload-processing"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954985    4909 cpu_manager.go:414] "containerMap: removing container" podUID="570ef14d-3b08-46d6-840d-7281aa3b6e21" containerName="payload-processing"
Nov 03 13:00:20.955095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.954990    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570ef14d-3b08-46d6-840d-7281aa3b6e21" containerName="payload-processing"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955130    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955138    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955147    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="36097829-e13a-4805-9a9e-e020c0b2fa4c" containerName="payload-processing"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955153    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955160    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955166    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955172    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="570ef14d-3b08-46d6-840d-7281aa3b6e21" containerName="payload-processing"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955179    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955184    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955191    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955198    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955203    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955209    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955215    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955222    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955228    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955233    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:20.955894 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.955239    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:20.958561 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:20.958544    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.013038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.013013    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"]
Nov 03 13:00:21.017876 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.017791    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8e61c0-6280-462c-9d6b-95665790b662-kube-api-access-mzc98\" (UniqueName: \"kubernetes.io/projected/4b8e61c0-6280-462c-9d6b-95665790b662-kube-api-access-mzc98\") pod \"4b8e61c0-6280-462c-9d6b-95665790b662\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") "
Nov 03 13:00:21.018800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.018000    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8e61c0-6280-462c-9d6b-95665790b662-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e61c0-6280-462c-9d6b-95665790b662-maas-api-service-ca\") pod \"4b8e61c0-6280-462c-9d6b-95665790b662\" (UID: \"4b8e61c0-6280-462c-9d6b-95665790b662\") "
Nov 03 13:00:21.018800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.018445    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.018800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.018535    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqwg\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-kube-api-access-jtqwg\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.018800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.018609    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.018800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.018752    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.019225 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.018925    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8e61c0-6280-462c-9d6b-95665790b662-maas-api-service-ca" pod "4b8e61c0-6280-462c-9d6b-95665790b662" (UID: "4b8e61c0-6280-462c-9d6b-95665790b662"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:21.019338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.019317    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e61c0-6280-462c-9d6b-95665790b662-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:21.021784 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.021226    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8e61c0-6280-462c-9d6b-95665790b662-kube-api-access-mzc98" pod "4b8e61c0-6280-462c-9d6b-95665790b662" (UID: "4b8e61c0-6280-462c-9d6b-95665790b662"). InnerVolumeSpecName "kube-api-access-mzc98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:21.120457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.120386    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqwg\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-kube-api-access-jtqwg\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.120457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.120428    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.120631 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.120505    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.120749 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.120728    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.120810 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.120797    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzc98\" (UniqueName: \"kubernetes.io/projected/4b8e61c0-6280-462c-9d6b-95665790b662-kube-api-access-mzc98\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:21.121250 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.121227    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.121335 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.121263    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.123437 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.123411    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.125565 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.125543    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" path="/var/lib/kubelet/pods/01030a13-b2a2-423b-aa1e-bf92c969edbd/volumes"
Nov 03 13:00:21.126019 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.126003    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a5c80b-3ed7-4009-8c26-bc262fb07651" path="/var/lib/kubelet/pods/44a5c80b-3ed7-4009-8c26-bc262fb07651/volumes"
Nov 03 13:00:21.126399 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.126386    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" path="/var/lib/kubelet/pods/5fab2913-3757-4544-b72e-8219fe9dac4d/volumes"
Nov 03 13:00:21.127120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.127104    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" path="/var/lib/kubelet/pods/79bbbd2b-c562-4575-9e20-6bf81c2ce99e/volumes"
Nov 03 13:00:21.127505 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.127491    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9602ddec-e761-4948-95f0-96427993446d" path="/var/lib/kubelet/pods/9602ddec-e761-4948-95f0-96427993446d/volumes"
Nov 03 13:00:21.172977 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.172953    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqwg\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-kube-api-access-jtqwg\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.272015 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.271986    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:21.279889 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.279865    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"]
Nov 03 13:00:21.559251 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.559222    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:21.570396 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.570371    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"
Nov 03 13:00:21.570532 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.570466    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg" event={"ID":"f243926b-8892-434a-afed-9e62b602c3ca","Type":"ContainerDied","Data":"711aad28520e8276d03b43a54e748debc6082ea8b79dd28eebb30c15160dc7b2"}
Nov 03 13:00:21.570602 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.570567    4909 scope.go:122] "RemoveContainer" containerID="48460781df8939890bf229c70ad95b1f4a17c44711458f1d7d718578f63ca89f"
Nov 03 13:00:21.572266 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.572140    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr" event={"ID":"4b8e61c0-6280-462c-9d6b-95665790b662","Type":"ContainerDied","Data":"6cb9c460a5ab0d2c0707b908d317d699d67267be3d415b13a1779735d9fbda29"}
Nov 03 13:00:21.572266 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.572155    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"
Nov 03 13:00:21.576638 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.576615    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5" event={"ID":"a5660732-c985-4aff-ae7f-a019ca46e893","Type":"ContainerDied","Data":"2d2c5a817bdc599f2cb45be38f60dbe9b8f4ca58e528e71da0e43d08ce8dca9c"}
Nov 03 13:00:21.576783 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.576662    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"
Nov 03 13:00:21.581761 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.581745    4909 scope.go:122] "RemoveContainer" containerID="0b40d31c70f496a48a1d123e957891ab56d39eda64381dce361872b06f26df21"
Nov 03 13:00:21.591416 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.591399    4909 scope.go:122] "RemoveContainer" containerID="5ef04c0ca72fac70130fc1582fc4eeb30077e1db52829593150cad889b560f42"
Nov 03 13:00:21.629497 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.629473    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f243926b-8892-434a-afed-9e62b602c3ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f243926b-8892-434a-afed-9e62b602c3ca-maas-api-service-ca\") pod \"f243926b-8892-434a-afed-9e62b602c3ca\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") "
Nov 03 13:00:21.629600 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.629585    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f243926b-8892-434a-afed-9e62b602c3ca-kube-api-access-hv6bf\" (UniqueName: \"kubernetes.io/projected/f243926b-8892-434a-afed-9e62b602c3ca-kube-api-access-hv6bf\") pod \"f243926b-8892-434a-afed-9e62b602c3ca\" (UID: \"f243926b-8892-434a-afed-9e62b602c3ca\") "
Nov 03 13:00:21.629835 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.629814    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f243926b-8892-434a-afed-9e62b602c3ca-maas-api-service-ca" pod "f243926b-8892-434a-afed-9e62b602c3ca" (UID: "f243926b-8892-434a-afed-9e62b602c3ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:21.630125 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.630107    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f243926b-8892-434a-afed-9e62b602c3ca-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:21.631670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.631649    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f243926b-8892-434a-afed-9e62b602c3ca-kube-api-access-hv6bf" pod "f243926b-8892-434a-afed-9e62b602c3ca" (UID: "f243926b-8892-434a-afed-9e62b602c3ca"). InnerVolumeSpecName "kube-api-access-hv6bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:21.654785 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.654763    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"]
Nov 03 13:00:21.655364 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:00:21.655337    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e8912a_6273_4c58_a3b1_62f19ac398f8.slice/crio-867d6165f26613f374d597bdc0072a2338085644ddcaef262c4ddf6a7d9a7eab WatchSource:0}: Error finding container 867d6165f26613f374d597bdc0072a2338085644ddcaef262c4ddf6a7d9a7eab: Status 404 returned error can't find the container with id 867d6165f26613f374d597bdc0072a2338085644ddcaef262c4ddf6a7d9a7eab
Nov 03 13:00:21.731227 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.731199    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv6bf\" (UniqueName: \"kubernetes.io/projected/f243926b-8892-434a-afed-9e62b602c3ca-kube-api-access-hv6bf\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:21.817824 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.817792    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"]
Nov 03 13:00:21.820749 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.820719    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895180-tbzn5"]
Nov 03 13:00:21.919250 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.919213    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"]
Nov 03 13:00:21.923422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:21.923398    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-532442-29895180-62xgr"]
Nov 03 13:00:22.024269 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:22.024194    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"]
Nov 03 13:00:22.027966 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:22.027937    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-aa9221-29895180-4qrsg"]
Nov 03 13:00:22.586446 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:22.586413    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" event={"ID":"d9e8912a-6273-4c58-a3b1-62f19ac398f8","Type":"ContainerStarted","Data":"e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549"}
Nov 03 13:00:22.586911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:22.586468    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" event={"ID":"d9e8912a-6273-4c58-a3b1-62f19ac398f8","Type":"ContainerStarted","Data":"867d6165f26613f374d597bdc0072a2338085644ddcaef262c4ddf6a7d9a7eab"}
Nov 03 13:00:22.586911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:22.586516    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:22.693496 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:22.693440    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" podStartSLOduration=2.693422363 podStartE2EDuration="2.693422363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:22.691611987 +0000 UTC m=+4171.704078663" watchObservedRunningTime="2026-11-03 13:00:22.693422363 +0000 UTC m=+4171.705889029"
Nov 03 13:00:23.124473 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:23.124445    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" path="/var/lib/kubelet/pods/4b8e61c0-6280-462c-9d6b-95665790b662/volumes"
Nov 03 13:00:23.124906 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:23.124892    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" path="/var/lib/kubelet/pods/a5660732-c985-4aff-ae7f-a019ca46e893/volumes"
Nov 03 13:00:23.125283 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:23.125271    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f243926b-8892-434a-afed-9e62b602c3ca" path="/var/lib/kubelet/pods/f243926b-8892-434a-afed-9e62b602c3ca/volumes"
Nov 03 13:00:33.596805 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.596719    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" podUID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:33.825312 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.825276    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"]
Nov 03 13:00:33.826587 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.826550    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" podUID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:33.852936 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.852864    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"]
Nov 03 13:00:33.853538 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853517    4909 cpu_manager.go:414] "containerMap: removing container" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853539    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bbbd2b-c562-4575-9e20-6bf81c2ce99e" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853559    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853566    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853588    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853595    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853605    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853613    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:00:33.853632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853627    4909 cpu_manager.go:414] "containerMap: removing container" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853635    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853647    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853654    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853667    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853674    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853683    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853690    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853933    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853949    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8e61c0-6280-462c-9d6b-95665790b662" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853961    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="01030a13-b2a2-423b-aa1e-bf92c969edbd" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853981    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5660732-c985-4aff-ae7f-a019ca46e893" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.853990    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9602ddec-e761-4948-95f0-96427993446d" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.854001    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fab2913-3757-4544-b72e-8219fe9dac4d" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.854014    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:00:33.854088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.854029    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:00:33.857525 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.857506    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:33.940612 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.940545    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"]
Nov 03 13:00:33.998554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.998522    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:33.998752 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.998601    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:33.998809 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.998769    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxsq\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-kube-api-access-kcxsq\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:33.998902 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:33.998887    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.100525 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.100496    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxsq\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-kube-api-access-kcxsq\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.100680 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.100573    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.100680 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.100617    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.100680 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.100668    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.101232 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.101209    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.101348 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.101327    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.103143 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.103093    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.152491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.152465    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxsq\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-kube-api-access-kcxsq\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.171324 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.171301    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.538648 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.538610    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"]
Nov 03 13:00:34.711780 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.711693    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" event={"ID":"93f22cbc-589d-4ecd-800e-c6b33dba9cb8","Type":"ContainerStarted","Data":"455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418"}
Nov 03 13:00:34.713407 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.713380    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:00:34.714331 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.714295    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" event={"ID":"93f22cbc-589d-4ecd-800e-c6b33dba9cb8","Type":"ContainerStarted","Data":"235085e856b162b31a96a75e356aa416115f5677269d525e4e1dd4bdcd310293"}
Nov 03 13:00:34.821488 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:34.821387    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" podStartSLOduration=1.8213734430000001 podStartE2EDuration="1.821373443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:34.820541624 +0000 UTC m=+4183.833008289" watchObservedRunningTime="2026-11-03 13:00:34.821373443 +0000 UTC m=+4183.833840105"
Nov 03 13:00:38.826032 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:38.825973    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" podUID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" containerName="payload-processing" containerID="cri-o://e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549" gracePeriod=25
Nov 03 13:00:39.157304 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.157273    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:39.272724 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.271317    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-routing-overlay-volume\") pod \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") "
Nov 03 13:00:39.272724 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.271444    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-kube-api-access-jtqwg\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-kube-api-access-jtqwg\") pod \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") "
Nov 03 13:00:39.272724 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.271489    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-plugins-config-volume\") pod \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") "
Nov 03 13:00:39.272724 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.271620    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-provider-credentials\") pod \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\" (UID: \"d9e8912a-6273-4c58-a3b1-62f19ac398f8\") "
Nov 03 13:00:39.277721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.273305    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-routing-overlay-volume" pod "d9e8912a-6273-4c58-a3b1-62f19ac398f8" (UID: "d9e8912a-6273-4c58-a3b1-62f19ac398f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:39.280214 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.280179    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-plugins-config-volume" pod "d9e8912a-6273-4c58-a3b1-62f19ac398f8" (UID: "d9e8912a-6273-4c58-a3b1-62f19ac398f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:39.281173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.281149    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-kube-api-access-jtqwg" pod "d9e8912a-6273-4c58-a3b1-62f19ac398f8" (UID: "d9e8912a-6273-4c58-a3b1-62f19ac398f8"). InnerVolumeSpecName "kube-api-access-jtqwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:39.282326 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.282301    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-provider-credentials" pod "d9e8912a-6273-4c58-a3b1-62f19ac398f8" (UID: "d9e8912a-6273-4c58-a3b1-62f19ac398f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:39.375105 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.374562    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:39.375105 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.374601    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:39.375105 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.374618    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtqwg\" (UniqueName: \"kubernetes.io/projected/d9e8912a-6273-4c58-a3b1-62f19ac398f8-kube-api-access-jtqwg\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:39.375105 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.374632    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e8912a-6273-4c58-a3b1-62f19ac398f8-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:39.564821 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:39.564782    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:00:39.747483 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.747443    4909 generic.go:356] "Generic (PLEG): container finished" podID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" containerID="e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549" exitCode=0
Nov 03 13:00:39.747686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.747515    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"
Nov 03 13:00:39.747686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.747530    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" event={"ID":"d9e8912a-6273-4c58-a3b1-62f19ac398f8","Type":"ContainerDied","Data":"e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549"}
Nov 03 13:00:39.747686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.747611    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn" event={"ID":"d9e8912a-6273-4c58-a3b1-62f19ac398f8","Type":"ContainerDied","Data":"867d6165f26613f374d597bdc0072a2338085644ddcaef262c4ddf6a7d9a7eab"}
Nov 03 13:00:39.747686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.747636    4909 scope.go:122] "RemoveContainer" containerID="e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549"
Nov 03 13:00:39.763593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.763577    4909 scope.go:122] "RemoveContainer" containerID="e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549"
Nov 03 13:00:39.764017 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:00:39.763994    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549\": container with ID starting with e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549 not found: ID does not exist" containerID="e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549"
Nov 03 13:00:39.764101 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.764024    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549"} err="failed to get container status \"e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549\": rpc error: code = NotFound desc = could not find container \"e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549\": container with ID starting with e25789c31ceac56c1ba6ce66db5639cd2de792bc4999900ebbbc6148dd3c1549 not found: ID does not exist"
Nov 03 13:00:39.899967 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.899937    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"]
Nov 03 13:00:39.945526 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:39.945494    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959scmfn"]
Nov 03 13:00:41.124074 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:41.124045    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" path="/var/lib/kubelet/pods/d9e8912a-6273-4c58-a3b1-62f19ac398f8/volumes"
Nov 03 13:00:46.638813 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:46.638777    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:00:46.729415 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:46.729372    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:51.673677 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:51.673646    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:00:51.688449 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:51.688422    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:00:51.691855 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:51.691825    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"]
Nov 03 13:00:56.692730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:56.692651    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6" podUID="3e41f9b2-8122-4835-9bcd-edd411486a7a" containerName="payload-pre-processing" containerID="cri-o://c801cfd4b132aa95bdf51b4bd79c32ce5fd2b3173fd4a9557ca3aa4d4f4e7a65" gracePeriod=25
Nov 03 13:00:56.727239 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:56.727211    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:56.864514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:56.864486    4909 generic.go:356] "Generic (PLEG): container finished" podID="3e41f9b2-8122-4835-9bcd-edd411486a7a" containerID="c801cfd4b132aa95bdf51b4bd79c32ce5fd2b3173fd4a9557ca3aa4d4f4e7a65" exitCode=0
Nov 03 13:00:56.864671 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:56.864567    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6" event={"ID":"3e41f9b2-8122-4835-9bcd-edd411486a7a","Type":"ContainerDied","Data":"c801cfd4b132aa95bdf51b4bd79c32ce5fd2b3173fd4a9557ca3aa4d4f4e7a65"}
Nov 03 13:00:57.002106 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.002071    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 13:00:57.133452 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.133419    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e41f9b2-8122-4835-9bcd-edd411486a7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e41f9b2-8122-4835-9bcd-edd411486a7a-plugins-config-volume\") pod \"3e41f9b2-8122-4835-9bcd-edd411486a7a\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") "
Nov 03 13:00:57.133611 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.133476    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e41f9b2-8122-4835-9bcd-edd411486a7a-kube-api-access-2s75q\" (UniqueName: \"kubernetes.io/projected/3e41f9b2-8122-4835-9bcd-edd411486a7a-kube-api-access-2s75q\") pod \"3e41f9b2-8122-4835-9bcd-edd411486a7a\" (UID: \"3e41f9b2-8122-4835-9bcd-edd411486a7a\") "
Nov 03 13:00:57.133907 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.133877    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e41f9b2-8122-4835-9bcd-edd411486a7a-plugins-config-volume" pod "3e41f9b2-8122-4835-9bcd-edd411486a7a" (UID: "3e41f9b2-8122-4835-9bcd-edd411486a7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:57.134094 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.134073    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e41f9b2-8122-4835-9bcd-edd411486a7a-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:57.136078 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.136040    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e41f9b2-8122-4835-9bcd-edd411486a7a-kube-api-access-2s75q" pod "3e41f9b2-8122-4835-9bcd-edd411486a7a" (UID: "3e41f9b2-8122-4835-9bcd-edd411486a7a"). InnerVolumeSpecName "kube-api-access-2s75q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:57.235827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.235767    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s75q\" (UniqueName: \"kubernetes.io/projected/3e41f9b2-8122-4835-9bcd-edd411486a7a-kube-api-access-2s75q\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:57.872484 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.872447    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"
Nov 03 13:00:57.872999 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.872484    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6" event={"ID":"3e41f9b2-8122-4835-9bcd-edd411486a7a","Type":"ContainerDied","Data":"e9ad48f8e76d00f6f1f321b844ecb2a4a4e17f18e7087ad06c0df047c99d183d"}
Nov 03 13:00:57.872999 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:57.872544    4909 scope.go:122] "RemoveContainer" containerID="c801cfd4b132aa95bdf51b4bd79c32ce5fd2b3173fd4a9557ca3aa4d4f4e7a65"
Nov 03 13:00:58.024607 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:58.024580    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"]
Nov 03 13:00:58.069987 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:58.069958    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d126a6-59bd4f4f5-7dzf6"]
Nov 03 13:00:59.123503 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:00:59.123463    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e41f9b2-8122-4835-9bcd-edd411486a7a" path="/var/lib/kubelet/pods/3e41f9b2-8122-4835-9bcd-edd411486a7a/volumes"
Nov 03 13:01:06.727051 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:06.727017    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:09.564801 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:09.564764    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:01:10.016532 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:10.016499    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1ab9c770bc22: secret "agc-cred-1ab9c770bc22" not found
Nov 03 13:01:10.016532 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:10.016529    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6e306ea651ee: secret "agc-cred-6e306ea651ee" not found
Nov 03 13:01:10.016821 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:10.016543    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b55d695-nzcgg: [secret "agc-cred-1ab9c770bc22" not found, secret "agc-cred-6e306ea651ee" not found]
Nov 03 13:01:10.016821 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:10.016606    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials podName:e3afccc9-9458-461e-ab3c-88780f5b1ce9 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:12.016592194 +0000 UTC m=+4341.029058839 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials") pod "payload-processing-external-model-599b55d695-nzcgg" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9") : [secret "agc-cred-1ab9c770bc22" not found, secret "agc-cred-6e306ea651ee" not found]
Nov 03 13:01:16.727474 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:16.727437    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:16.790247 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:16.790216    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"]
Nov 03 13:01:16.791644 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:16.791604    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:16.829590 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:16.829562    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"]
Nov 03 13:01:19.973670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:19.973638    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"]
Nov 03 13:01:19.974173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:19.973888    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" containerID="cri-o://0e80d0f24b4815a6784abbc1cd9d92d4125e5ce5d00e942e444f83aa4bda89a7" gracePeriod=30
Nov 03 13:01:21.607561 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:21.607526    4909 patch_prober.go:29] interesting pod/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:01:21.607970 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:21.607580    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:01:21.791515 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:21.791480    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing" containerID="cri-o://455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418" gracePeriod=25
Nov 03 13:01:21.860626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:21.836749    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg" podUID="3f5a6b84-7986-4f26-8708-b4922cf636c8" containerName="payload-processing" containerID="cri-o://463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054" gracePeriod=25
Nov 03 13:01:22.175715 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.175678    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:01:22.179372 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.179342    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 13:01:22.179672 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.179650    4909 generic.go:356] "Generic (PLEG): container finished" podID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerID="455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418" exitCode=0
Nov 03 13:01:22.179784 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.179685    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"
Nov 03 13:01:22.179784 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.179744    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" event={"ID":"93f22cbc-589d-4ecd-800e-c6b33dba9cb8","Type":"ContainerDied","Data":"455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418"}
Nov 03 13:01:22.179896 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.179789    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m" event={"ID":"93f22cbc-589d-4ecd-800e-c6b33dba9cb8","Type":"ContainerDied","Data":"235085e856b162b31a96a75e356aa416115f5677269d525e4e1dd4bdcd310293"}
Nov 03 13:01:22.179896 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.179811    4909 scope.go:122] "RemoveContainer" containerID="455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418"
Nov 03 13:01:22.180982 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.180965    4909 generic.go:356] "Generic (PLEG): container finished" podID="3f5a6b84-7986-4f26-8708-b4922cf636c8" containerID="463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054" exitCode=0
Nov 03 13:01:22.181083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.181003    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"
Nov 03 13:01:22.181083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.181008    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg" event={"ID":"3f5a6b84-7986-4f26-8708-b4922cf636c8","Type":"ContainerDied","Data":"463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054"}
Nov 03 13:01:22.181176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.181101    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg" event={"ID":"3f5a6b84-7986-4f26-8708-b4922cf636c8","Type":"ContainerDied","Data":"95a2a464781f1ee528155443af8c9cb72f071634e5cf50e4bf591023aab465a1"}
Nov 03 13:01:22.188995 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.188981    4909 scope.go:122] "RemoveContainer" containerID="455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418"
Nov 03 13:01:22.189258 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:22.189240    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418\": container with ID starting with 455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418 not found: ID does not exist" containerID="455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418"
Nov 03 13:01:22.189303 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.189264    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418"} err="failed to get container status \"455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418\": rpc error: code = NotFound desc = could not find container \"455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418\": container with ID starting with 455fe18cdb83ca20677caa383fb4d223c2ad72649871ad4275440c312fa80418 not found: ID does not exist"
Nov 03 13:01:22.189303 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.189281    4909 scope.go:122] "RemoveContainer" containerID="463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054"
Nov 03 13:01:22.198918 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.198898    4909 scope.go:122] "RemoveContainer" containerID="463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054"
Nov 03 13:01:22.199149 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:22.199132    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054\": container with ID starting with 463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054 not found: ID does not exist" containerID="463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054"
Nov 03 13:01:22.199200 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.199155    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054"} err="failed to get container status \"463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054\": rpc error: code = NotFound desc = could not find container \"463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054\": container with ID starting with 463b0f3c69dad21a3114455b1acb5db36874215a9b145517fe1f3aecf2622054 not found: ID does not exist"
Nov 03 13:01:22.277378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277354    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-plugins-config-volume\") pod \"3f5a6b84-7986-4f26-8708-b4922cf636c8\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") "
Nov 03 13:01:22.277478 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277428    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-provider-credentials\") pod \"3f5a6b84-7986-4f26-8708-b4922cf636c8\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") "
Nov 03 13:01:22.277529 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277496    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-kube-api-access-kcxsq\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-kube-api-access-kcxsq\") pod \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") "
Nov 03 13:01:22.277570 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277531    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-routing-overlay-volume\") pod \"3f5a6b84-7986-4f26-8708-b4922cf636c8\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") "
Nov 03 13:01:22.277607 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277579    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-plugins-config-volume\") pod \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") "
Nov 03 13:01:22.277655 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277619    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-kube-api-access-h7799\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-kube-api-access-h7799\") pod \"3f5a6b84-7986-4f26-8708-b4922cf636c8\" (UID: \"3f5a6b84-7986-4f26-8708-b4922cf636c8\") "
Nov 03 13:01:22.277739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277676    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-routing-overlay-volume\") pod \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") "
Nov 03 13:01:22.277815 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277788    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-provider-credentials\") pod \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\" (UID: \"93f22cbc-589d-4ecd-800e-c6b33dba9cb8\") "
Nov 03 13:01:22.277946 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.277916    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-routing-overlay-volume" pod "3f5a6b84-7986-4f26-8708-b4922cf636c8" (UID: "3f5a6b84-7986-4f26-8708-b4922cf636c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:22.278148 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.278122    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-plugins-config-volume" pod "93f22cbc-589d-4ecd-800e-c6b33dba9cb8" (UID: "93f22cbc-589d-4ecd-800e-c6b33dba9cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:22.278234 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.278160    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-routing-overlay-volume" pod "93f22cbc-589d-4ecd-800e-c6b33dba9cb8" (UID: "93f22cbc-589d-4ecd-800e-c6b33dba9cb8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:22.278431 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.278408    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.278506 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.278438    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.278506 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.278444    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-plugins-config-volume" pod "3f5a6b84-7986-4f26-8708-b4922cf636c8" (UID: "3f5a6b84-7986-4f26-8708-b4922cf636c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:22.278506 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.278454    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.279849 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.279822    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-kube-api-access-kcxsq" pod "93f22cbc-589d-4ecd-800e-c6b33dba9cb8" (UID: "93f22cbc-589d-4ecd-800e-c6b33dba9cb8"). InnerVolumeSpecName "kube-api-access-kcxsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:22.279923 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.279859    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-kube-api-access-h7799" pod "3f5a6b84-7986-4f26-8708-b4922cf636c8" (UID: "3f5a6b84-7986-4f26-8708-b4922cf636c8"). InnerVolumeSpecName "kube-api-access-h7799". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:22.279923 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.279866    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-provider-credentials" pod "3f5a6b84-7986-4f26-8708-b4922cf636c8" (UID: "3f5a6b84-7986-4f26-8708-b4922cf636c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:22.280045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.280032    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-provider-credentials" pod "93f22cbc-589d-4ecd-800e-c6b33dba9cb8" (UID: "93f22cbc-589d-4ecd-800e-c6b33dba9cb8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:22.379572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.379504    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.379572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.379531    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxsq\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-kube-api-access-kcxsq\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.379572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.379547    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7799\" (UniqueName: \"kubernetes.io/projected/3f5a6b84-7986-4f26-8708-b4922cf636c8-kube-api-access-h7799\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.379572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.379556    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93f22cbc-589d-4ecd-800e-c6b33dba9cb8-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.379572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.379565    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5a6b84-7986-4f26-8708-b4922cf636c8-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:22.596672 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.596642    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"]
Nov 03 13:01:22.601759 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.601727    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-855bkc7rg"]
Nov 03 13:01:22.703591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.703558    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"]
Nov 03 13:01:22.707554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:22.707529    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffjwb2m"]
Nov 03 13:01:23.124457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:23.124383    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5a6b84-7986-4f26-8708-b4922cf636c8" path="/var/lib/kubelet/pods/3f5a6b84-7986-4f26-8708-b4922cf636c8/volumes"
Nov 03 13:01:23.124837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:23.124822    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" path="/var/lib/kubelet/pods/93f22cbc-589d-4ecd-800e-c6b33dba9cb8/volumes"
Nov 03 13:01:25.071579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.071549    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"]
Nov 03 13:01:25.071990 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.071781    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy" containerID="cri-o://f1f7e3e2158dbbc1a756c5eeb004d311471ad1039f1456c1a1363a5ee357f038" gracePeriod=30
Nov 03 13:01:25.301751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.301655    4909 generic.go:356] "Generic (PLEG): container finished" podID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerID="0e80d0f24b4815a6784abbc1cd9d92d4125e5ce5d00e942e444f83aa4bda89a7" exitCode=0
Nov 03 13:01:25.302217 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.302191    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" event={"ID":"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2","Type":"ContainerDied","Data":"0e80d0f24b4815a6784abbc1cd9d92d4125e5ce5d00e942e444f83aa4bda89a7"}
Nov 03 13:01:25.302446 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.302413    4909 scope.go:122] "RemoveContainer" containerID="82842d7e672a6524c760ca88d152b7453aabc92e8170c64a603f84ce29420a06"
Nov 03 13:01:25.378971 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.378946    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 13:01:25.451516 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451489    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-envoy\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.451632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451599    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-certs\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.451691 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451676    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-podinfo\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.451783 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451743    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-token\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-token\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.451852 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451818    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-kube-api-access-4vbxt\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-kube-api-access-4vbxt\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.451911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451849    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istiod-ca-cert\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.451911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451867    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-certs" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:25.451911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451881    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-socket\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.452115 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.451950    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-credential-socket\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.452115 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452021    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-data\") pod \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\" (UID: \"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2\") "
Nov 03 13:01:25.452272 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452244    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-socket" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:25.452343 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452267    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istiod-ca-cert" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:25.452435 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452415    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-credential-socket" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:25.452497 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452457    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-data" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:25.452559 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452525    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.452559 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452542    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.452559 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452554    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.452765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452567    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.452765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.452576    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.454115 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.454094    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-kube-api-access-4vbxt" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "kube-api-access-4vbxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:25.454207 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.454121    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-token" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:25.454329 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.454304    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-envoy" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:25.454424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.454325    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-podinfo" pod "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" (UID: "93edf994-9d3f-4af0-a998-6c8a2cb9a2c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:01:25.553558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.553490    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.553558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.553517    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.553558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.553527    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:25.553558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:25.553537    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbxt\" (UniqueName: \"kubernetes.io/projected/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2-kube-api-access-4vbxt\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:26.309400 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:26.309368    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"
Nov 03 13:01:26.309837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:26.309403    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk" event={"ID":"93edf994-9d3f-4af0-a998-6c8a2cb9a2c2","Type":"ContainerDied","Data":"2b0a951e72ad8304c6f5c34e26c504b26079fc58fd49bcce399dbbab556554cf"}
Nov 03 13:01:26.309837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:26.309463    4909 scope.go:122] "RemoveContainer" containerID="0e80d0f24b4815a6784abbc1cd9d92d4125e5ce5d00e942e444f83aa4bda89a7"
Nov 03 13:01:26.507572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:26.507537    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"]
Nov 03 13:01:26.510351 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:26.510326    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b737fe-openshift-default-5478d75849-z7vtk"]
Nov 03 13:01:27.124680 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:27.124646    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" path="/var/lib/kubelet/pods/93edf994-9d3f-4af0-a998-6c8a2cb9a2c2/volumes"
Nov 03 13:01:27.237600 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:27.237570    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"]
Nov 03 13:01:27.336794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:27.336763    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"]
Nov 03 13:01:30.338026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.337994    4909 generic.go:356] "Generic (PLEG): container finished" podID="b73f9e32-8924-4f74-8623-00be96ff490b" containerID="f1f7e3e2158dbbc1a756c5eeb004d311471ad1039f1456c1a1363a5ee357f038" exitCode=0
Nov 03 13:01:30.338401 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.338073    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" event={"ID":"b73f9e32-8924-4f74-8623-00be96ff490b","Type":"ContainerDied","Data":"f1f7e3e2158dbbc1a756c5eeb004d311471ad1039f1456c1a1363a5ee357f038"}
Nov 03 13:01:30.338401 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.338136    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk" event={"ID":"b73f9e32-8924-4f74-8623-00be96ff490b","Type":"ContainerDied","Data":"840ad01132615514d2c058e0630360198146f96ff4c5df872eb46cab44693435"}
Nov 03 13:01:30.338401 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.338163    4909 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840ad01132615514d2c058e0630360198146f96ff4c5df872eb46cab44693435"
Nov 03 13:01:30.338401 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.338178    4909 scope.go:122] "RemoveContainer" containerID="8d8e9a4dbf3861a82d056dc853655b380f67017aede09dab51e75dd9de84161a"
Nov 03 13:01:30.358195 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.358178    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 13:01:30.432006 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.431980    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-socket\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432146 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432019    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-certs\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432146 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432078    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-data\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432275 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432169    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b73f9e32-8924-4f74-8623-00be96ff490b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f9e32-8924-4f74-8623-00be96ff490b-istio-podinfo\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432275 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432225    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-istio-token\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-istio-token\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432275 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432264    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-kube-api-access-xgngz\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-kube-api-access-xgngz\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432224    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-socket" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:30.432444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432305    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-credential-socket\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432341    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-envoy\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432358    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-certs" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:30.432444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432428    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-data" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:30.432663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432430    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73f9e32-8924-4f74-8623-00be96ff490b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f9e32-8924-4f74-8623-00be96ff490b-istiod-ca-cert\") pod \"b73f9e32-8924-4f74-8623-00be96ff490b\" (UID: \"b73f9e32-8924-4f74-8623-00be96ff490b\") "
Nov 03 13:01:30.432802 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432781    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73f9e32-8924-4f74-8623-00be96ff490b-istiod-ca-cert" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:30.432875 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.432810    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-credential-socket" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:30.433067 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.433046    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.433153 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.433073    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.433153 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.433090    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f9e32-8924-4f74-8623-00be96ff490b-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.433153 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.433105    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.433153 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.433119    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.434428 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.434400    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-istio-token" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:30.434547 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.434512    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b73f9e32-8924-4f74-8623-00be96ff490b-istio-podinfo" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:01:30.434547 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.434521    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-envoy" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:30.434953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.434934    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-kube-api-access-xgngz" pod "b73f9e32-8924-4f74-8623-00be96ff490b" (UID: "b73f9e32-8924-4f74-8623-00be96ff490b"). InnerVolumeSpecName "kube-api-access-xgngz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:30.534409 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.534379    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f9e32-8924-4f74-8623-00be96ff490b-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.534409 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.534406    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.534409 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.534416    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgngz\" (UniqueName: \"kubernetes.io/projected/b73f9e32-8924-4f74-8623-00be96ff490b-kube-api-access-xgngz\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.534734 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.534425    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f9e32-8924-4f74-8623-00be96ff490b-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:30.563744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.562556    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"]
Nov 03 13:01:30.564309 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564281    4909 cpu_manager.go:414] "containerMap: removing container" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy"
Nov 03 13:01:30.564430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564312    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy"
Nov 03 13:01:30.564430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564329    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy"
Nov 03 13:01:30.564430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564338    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy"
Nov 03 13:01:30.564430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564355    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:01:30.564430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564362    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f243926b-8892-434a-afed-9e62b602c3ca" containerName="cleanup"
Nov 03 13:01:30.564430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564398    4909 cpu_manager.go:414] "containerMap: removing container" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing"
Nov 03 13:01:30.564430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564407    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564462    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3f5a6b84-7986-4f26-8708-b4922cf636c8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564470    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5a6b84-7986-4f26-8708-b4922cf636c8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564480    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564488    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564500    4909 cpu_manager.go:414] "containerMap: removing container" podUID="3e41f9b2-8122-4835-9bcd-edd411486a7a" containerName="payload-pre-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564509    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e41f9b2-8122-4835-9bcd-edd411486a7a" containerName="payload-pre-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564656    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564670    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564685    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f22cbc-589d-4ecd-800e-c6b33dba9cb8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564725    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5a6b84-7986-4f26-8708-b4922cf636c8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564736    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e8912a-6273-4c58-a3b1-62f19ac398f8" containerName="payload-processing"
Nov 03 13:01:30.564840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.564745    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e41f9b2-8122-4835-9bcd-edd411486a7a" containerName="payload-pre-processing"
Nov 03 13:01:30.625777 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.612103    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.699581    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"]
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700096    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-certs\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700204    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-credential-socket\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700232    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-podinfo\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700266    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-token\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700485    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsncw\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-kube-api-access-hsncw\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700536    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-socket\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700613    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-data\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700679    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a09fd-6d12-43f2-8672-815f5f4d7863-istiod-ca-cert\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.733023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.700731    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-envoy\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.774786 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.697826    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d4609f-openshift-default-dockercfg-t9np5\""
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802272    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a09fd-6d12-43f2-8672-815f5f4d7863-istiod-ca-cert\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802330    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-envoy\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802376    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-certs\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802470    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-credential-socket\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802499    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-podinfo\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802532    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-token\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802673    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsncw\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-kube-api-access-hsncw\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802733    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-socket\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.802812    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-data\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.803406    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-data\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.813847    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-socket\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.817019    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-podinfo\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.817302    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-certs\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.821176 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.817546    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-credential-socket\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.828282 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.828245    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a09fd-6d12-43f2-8672-815f5f4d7863-istiod-ca-cert\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.828471 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.828385    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-envoy\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.866017 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.865989    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-token\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:30.866272 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:30.866250    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsncw\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-kube-api-access-hsncw\") pod \"e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") " pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:31.130744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.130110    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:31.347073 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.347041    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"
Nov 03 13:01:31.560838 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.560779    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"]
Nov 03 13:01:31.564024 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.563997    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-532442-openshift-default-79bf5f49c5-wwztk"]
Nov 03 13:01:31.598052 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.598028    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"]
Nov 03 13:01:31.627279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.614545    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:01:31.627279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.614629    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:01:31.627279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:31.614672    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:01:32.237991 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.237954    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59" podUID="b91c14c7-3143-4427-a746-6758a9ac1170" containerName="payload-processing" containerID="cri-o://f02a9661d6d32a80fed935b26ca4a50824d0ff8f2a0e52cf6782db45b2af7a37" gracePeriod=25
Nov 03 13:01:32.337413 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.337365    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl" podUID="ca135c32-5abe-4295-8985-990029dfa602" containerName="payload-pre-processing" containerID="cri-o://79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888" gracePeriod=25
Nov 03 13:01:32.356674 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.356617    4909 generic.go:356] "Generic (PLEG): container finished" podID="b91c14c7-3143-4427-a746-6758a9ac1170" containerID="f02a9661d6d32a80fed935b26ca4a50824d0ff8f2a0e52cf6782db45b2af7a37" exitCode=0
Nov 03 13:01:32.357047 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.356686    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59" event={"ID":"b91c14c7-3143-4427-a746-6758a9ac1170","Type":"ContainerDied","Data":"f02a9661d6d32a80fed935b26ca4a50824d0ff8f2a0e52cf6782db45b2af7a37"}
Nov 03 13:01:32.358488 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.358455    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" event={"ID":"283a09fd-6d12-43f2-8672-815f5f4d7863","Type":"ContainerStarted","Data":"8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43"}
Nov 03 13:01:32.358577 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.358503    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" event={"ID":"283a09fd-6d12-43f2-8672-815f5f4d7863","Type":"ContainerStarted","Data":"b2713c56b3d57402b8ae8fecad24b983b4e71ad7d421fccbe8aa185a5e21d631"}
Nov 03 13:01:32.472011 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.471951    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" podStartSLOduration=2.471917727 podStartE2EDuration="2.471917727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:01:32.465385701 +0000 UTC m=+4241.477852365" watchObservedRunningTime="2026-11-03 13:01:32.471917727 +0000 UTC m=+4241.484384391"
Nov 03 13:01:32.516151 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.516122    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 13:01:32.636493 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.636447    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91c14c7-3143-4427-a746-6758a9ac1170-kube-api-access-q2gwn\" (UniqueName: \"kubernetes.io/projected/b91c14c7-3143-4427-a746-6758a9ac1170-kube-api-access-q2gwn\") pod \"b91c14c7-3143-4427-a746-6758a9ac1170\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") "
Nov 03 13:01:32.636727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.636604    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91c14c7-3143-4427-a746-6758a9ac1170-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91c14c7-3143-4427-a746-6758a9ac1170-plugins-config-volume\") pod \"b91c14c7-3143-4427-a746-6758a9ac1170\" (UID: \"b91c14c7-3143-4427-a746-6758a9ac1170\") "
Nov 03 13:01:32.639094 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.637956    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91c14c7-3143-4427-a746-6758a9ac1170-plugins-config-volume" pod "b91c14c7-3143-4427-a746-6758a9ac1170" (UID: "b91c14c7-3143-4427-a746-6758a9ac1170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:32.649866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.641418    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91c14c7-3143-4427-a746-6758a9ac1170-kube-api-access-q2gwn" pod "b91c14c7-3143-4427-a746-6758a9ac1170" (UID: "b91c14c7-3143-4427-a746-6758a9ac1170"). InnerVolumeSpecName "kube-api-access-q2gwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:32.665476 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.665451    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 13:01:32.738194 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.738160    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91c14c7-3143-4427-a746-6758a9ac1170-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:32.738194 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.738184    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2gwn\" (UniqueName: \"kubernetes.io/projected/b91c14c7-3143-4427-a746-6758a9ac1170-kube-api-access-q2gwn\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:32.839257 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.839175    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca135c32-5abe-4295-8985-990029dfa602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca135c32-5abe-4295-8985-990029dfa602-plugins-config-volume\") pod \"ca135c32-5abe-4295-8985-990029dfa602\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") "
Nov 03 13:01:32.839257 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.839244    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca135c32-5abe-4295-8985-990029dfa602-kube-api-access-2h2z4\" (UniqueName: \"kubernetes.io/projected/ca135c32-5abe-4295-8985-990029dfa602-kube-api-access-2h2z4\") pod \"ca135c32-5abe-4295-8985-990029dfa602\" (UID: \"ca135c32-5abe-4295-8985-990029dfa602\") "
Nov 03 13:01:32.839634 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.839608    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca135c32-5abe-4295-8985-990029dfa602-plugins-config-volume" pod "ca135c32-5abe-4295-8985-990029dfa602" (UID: "ca135c32-5abe-4295-8985-990029dfa602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:32.841318 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.841296    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca135c32-5abe-4295-8985-990029dfa602-kube-api-access-2h2z4" pod "ca135c32-5abe-4295-8985-990029dfa602" (UID: "ca135c32-5abe-4295-8985-990029dfa602"). InnerVolumeSpecName "kube-api-access-2h2z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:32.941300 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.941272    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca135c32-5abe-4295-8985-990029dfa602-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:32.941300 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:32.941296    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2z4\" (UniqueName: \"kubernetes.io/projected/ca135c32-5abe-4295-8985-990029dfa602-kube-api-access-2h2z4\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:33.124524 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.124449    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" path="/var/lib/kubelet/pods/b73f9e32-8924-4f74-8623-00be96ff490b/volumes"
Nov 03 13:01:33.130278 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.130254    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:33.136317 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.136298    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:33.366343 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.366312    4909 generic.go:356] "Generic (PLEG): container finished" podID="ca135c32-5abe-4295-8985-990029dfa602" containerID="79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888" exitCode=0
Nov 03 13:01:33.366792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.366358    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"
Nov 03 13:01:33.366792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.366389    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl" event={"ID":"ca135c32-5abe-4295-8985-990029dfa602","Type":"ContainerDied","Data":"79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888"}
Nov 03 13:01:33.366792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.366443    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl" event={"ID":"ca135c32-5abe-4295-8985-990029dfa602","Type":"ContainerDied","Data":"6d7a1f9bee631a0b85a4feed3409e069d54f90b2976d49411eb1ec3028933879"}
Nov 03 13:01:33.366792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.366466    4909 scope.go:122] "RemoveContainer" containerID="79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888"
Nov 03 13:01:33.368120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.368096    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59" event={"ID":"b91c14c7-3143-4427-a746-6758a9ac1170","Type":"ContainerDied","Data":"a4e3666a57a009e49a3b5544e6eb6a034b440fbb3eda90c808a75ee4f2dabeb3"}
Nov 03 13:01:33.368120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.368107    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"
Nov 03 13:01:33.368795 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.368725    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:33.369916 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.369885    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:01:33.378166 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.378094    4909 scope.go:122] "RemoveContainer" containerID="79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888"
Nov 03 13:01:33.378343 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:33.378323    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888\": container with ID starting with 79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888 not found: ID does not exist" containerID="79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888"
Nov 03 13:01:33.378424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.378350    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888"} err="failed to get container status \"79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888\": rpc error: code = NotFound desc = could not find container \"79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888\": container with ID starting with 79053a53018f5e8f51c94b43f586645132d4da9643ea9044634fd2c98144e888 not found: ID does not exist"
Nov 03 13:01:33.378424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.378370    4909 scope.go:122] "RemoveContainer" containerID="f02a9661d6d32a80fed935b26ca4a50824d0ff8f2a0e52cf6782db45b2af7a37"
Nov 03 13:01:33.517897 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.517869    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"]
Nov 03 13:01:33.523156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.523133    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb74b5-9c6844c79-qfxtl"]
Nov 03 13:01:33.622905 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.622879    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"]
Nov 03 13:01:33.629827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:33.629757    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb74b5-544c9658f7-bbc59"]
Nov 03 13:01:35.123824 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:35.123797    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91c14c7-3143-4427-a746-6758a9ac1170" path="/var/lib/kubelet/pods/b91c14c7-3143-4427-a746-6758a9ac1170/volumes"
Nov 03 13:01:35.124201 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:35.124189    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca135c32-5abe-4295-8985-990029dfa602" path="/var/lib/kubelet/pods/ca135c32-5abe-4295-8985-990029dfa602/volumes"
Nov 03 13:01:39.322790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.322752    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"]
Nov 03 13:01:39.323370 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323349    4909 cpu_manager.go:414] "containerMap: removing container" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy"
Nov 03 13:01:39.323422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323371    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy"
Nov 03 13:01:39.323422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323383    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy"
Nov 03 13:01:39.323422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323388    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy"
Nov 03 13:01:39.323422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323401    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ca135c32-5abe-4295-8985-990029dfa602" containerName="payload-pre-processing"
Nov 03 13:01:39.323422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323407    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca135c32-5abe-4295-8985-990029dfa602" containerName="payload-pre-processing"
Nov 03 13:01:39.323633 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323429    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b91c14c7-3143-4427-a746-6758a9ac1170" containerName="payload-processing"
Nov 03 13:01:39.323633 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323434    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91c14c7-3143-4427-a746-6758a9ac1170" containerName="payload-processing"
Nov 03 13:01:39.323633 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323536    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="93edf994-9d3f-4af0-a998-6c8a2cb9a2c2" containerName="istio-proxy"
Nov 03 13:01:39.323633 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323545    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73f9e32-8924-4f74-8623-00be96ff490b" containerName="istio-proxy"
Nov 03 13:01:39.323633 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323554    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca135c32-5abe-4295-8985-990029dfa602" containerName="payload-pre-processing"
Nov 03 13:01:39.323633 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.323564    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91c14c7-3143-4427-a746-6758a9ac1170" containerName="payload-processing"
Nov 03 13:01:39.328715 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.328685    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:39.387722 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.381456    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"]
Nov 03 13:01:39.453721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.452813    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d4609f\""
Nov 03 13:01:39.453721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.453462    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d4609f-dockercfg-5nkrw\""
Nov 03 13:01:39.507039 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.498631    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"]
Nov 03 13:01:39.507039 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.504072    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.552134 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.552101    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"]
Nov 03 13:01:39.561719 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.561668    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.561952 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.561926    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e11887-1355-4bad-b890-3a32ab9edf74-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:39.562071 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.562054    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwnf\" (UniqueName: \"kubernetes.io/projected/10e11887-1355-4bad-b890-3a32ab9edf74-kube-api-access-dvwnf\") pod \"payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:39.562245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.562230    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7psnd\" (UniqueName: \"kubernetes.io/projected/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-kube-api-access-7psnd\") pod \"payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.567758 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:01:39.567734    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:01:39.663917 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.663845    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e11887-1355-4bad-b890-3a32ab9edf74-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:39.663917 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.663880    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwnf\" (UniqueName: \"kubernetes.io/projected/10e11887-1355-4bad-b890-3a32ab9edf74-kube-api-access-dvwnf\") pod \"payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:39.664101 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.663949    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7psnd\" (UniqueName: \"kubernetes.io/projected/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-kube-api-access-7psnd\") pod \"payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.664242 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.664219    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.664667 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.664649    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e11887-1355-4bad-b890-3a32ab9edf74-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:39.664879 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.664860    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.716369 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.716336    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7psnd\" (UniqueName: \"kubernetes.io/projected/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-kube-api-access-7psnd\") pod \"payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.716518 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.716452    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwnf\" (UniqueName: \"kubernetes.io/projected/10e11887-1355-4bad-b890-3a32ab9edf74-kube-api-access-dvwnf\") pod \"payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:39.833950 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.833923    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:39.942101 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:39.942069    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:40.202461 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.202431    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"]
Nov 03 13:01:40.204079 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:01:40.204051    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f1b5ed_c0aa_493e_8d31_821d4fc115a2.slice/crio-527ddfc3dbee1b8115bc6f44d877d2f072c5ebe3176e428cab6aa901a8772f46 WatchSource:0}: Error finding container 527ddfc3dbee1b8115bc6f44d877d2f072c5ebe3176e428cab6aa901a8772f46: Status 404 returned error can't find the container with id 527ddfc3dbee1b8115bc6f44d877d2f072c5ebe3176e428cab6aa901a8772f46
Nov 03 13:01:40.316276 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.316250    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"]
Nov 03 13:01:40.318020 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:01:40.317997    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e11887_1355_4bad_b890_3a32ab9edf74.slice/crio-b5eb8521e856528cf2b6bfe0e4946a4d36372940b6a0b69e26da8560a10d8aed WatchSource:0}: Error finding container b5eb8521e856528cf2b6bfe0e4946a4d36372940b6a0b69e26da8560a10d8aed: Status 404 returned error can't find the container with id b5eb8521e856528cf2b6bfe0e4946a4d36372940b6a0b69e26da8560a10d8aed
Nov 03 13:01:40.478473 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.478370    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" event={"ID":"10e11887-1355-4bad-b890-3a32ab9edf74","Type":"ContainerStarted","Data":"6bebfccfe04710fdd2b6ff3062917cb6a97127a5b9c265046a00d6dc726a1fe5"}
Nov 03 13:01:40.478473 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.478440    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" event={"ID":"10e11887-1355-4bad-b890-3a32ab9edf74","Type":"ContainerStarted","Data":"b5eb8521e856528cf2b6bfe0e4946a4d36372940b6a0b69e26da8560a10d8aed"}
Nov 03 13:01:40.478937 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.478581    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:40.479977 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.479953    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx" event={"ID":"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2","Type":"ContainerStarted","Data":"b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8"}
Nov 03 13:01:40.480083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.479990    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx" event={"ID":"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2","Type":"ContainerStarted","Data":"527ddfc3dbee1b8115bc6f44d877d2f072c5ebe3176e428cab6aa901a8772f46"}
Nov 03 13:01:40.480083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.480020    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:40.587228 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.587176    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" podStartSLOduration=1.58716006 podStartE2EDuration="1.58716006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:01:40.583954558 +0000 UTC m=+4249.596421225" watchObservedRunningTime="2026-11-03 13:01:40.58716006 +0000 UTC m=+4249.599626723"
Nov 03 13:01:40.691657 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:40.691604    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx" podStartSLOduration=1.691591984 podStartE2EDuration="1.691591984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:01:40.688118899 +0000 UTC m=+4249.700585563" watchObservedRunningTime="2026-11-03 13:01:40.691591984 +0000 UTC m=+4249.704058646"
Nov 03 13:01:43.432074 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:43.432003    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:01:51.489962 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:51.489931    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:01:51.490439 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:51.490052    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:01:56.254975 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:01:56.254943    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"]
Nov 03 13:02:01.256189 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.256086    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg" podUID="e3afccc9-9458-461e-ab3c-88780f5b1ce9" containerName="payload-processing" containerID="cri-o://e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83" gracePeriod=25
Nov 03 13:02:01.584084 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.584046    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 13:02:01.629426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.629391    4909 generic.go:356] "Generic (PLEG): container finished" podID="e3afccc9-9458-461e-ab3c-88780f5b1ce9" containerID="e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83" exitCode=0
Nov 03 13:02:01.629572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.629441    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"
Nov 03 13:02:01.629572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.629473    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg" event={"ID":"e3afccc9-9458-461e-ab3c-88780f5b1ce9","Type":"ContainerDied","Data":"e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83"}
Nov 03 13:02:01.629572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.629526    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b55d695-nzcgg" event={"ID":"e3afccc9-9458-461e-ab3c-88780f5b1ce9","Type":"ContainerDied","Data":"720b42d7fa9c2a5ca0076ddaf353c54f25027465c0173cd130612eb23ce7f167"}
Nov 03 13:02:01.629572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.629555    4909 scope.go:122] "RemoveContainer" containerID="e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83"
Nov 03 13:02:01.639319 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.639292    4909 scope.go:122] "RemoveContainer" containerID="e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83"
Nov 03 13:02:01.639576 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:01.639555    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83\": container with ID starting with e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83 not found: ID does not exist" containerID="e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83"
Nov 03 13:02:01.639634 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.639583    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83"} err="failed to get container status \"e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83\": rpc error: code = NotFound desc = could not find container \"e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83\": container with ID starting with e9620d70a5cdbd1ee942321a45f6d4c2b246315d0a9d8db12b49ffc03f610e83 not found: ID does not exist"
Nov 03 13:02:01.646071 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646052    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-routing-overlay-volume\") pod \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") "
Nov 03 13:02:01.646143 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646089    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-kube-api-access-zz8nj\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-kube-api-access-zz8nj\") pod \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") "
Nov 03 13:02:01.646180 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646140    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-plugins-config-volume\") pod \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") "
Nov 03 13:02:01.646314 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646292    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials\") pod \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\" (UID: \"e3afccc9-9458-461e-ab3c-88780f5b1ce9\") "
Nov 03 13:02:01.646410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646392    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-routing-overlay-volume" pod "e3afccc9-9458-461e-ab3c-88780f5b1ce9" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:01.646518 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646498    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-plugins-config-volume" pod "e3afccc9-9458-461e-ab3c-88780f5b1ce9" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:01.646798 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646779    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:01.646855 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.646803    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3afccc9-9458-461e-ab3c-88780f5b1ce9-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:01.648198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.648172    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-kube-api-access-zz8nj" pod "e3afccc9-9458-461e-ab3c-88780f5b1ce9" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9"). InnerVolumeSpecName "kube-api-access-zz8nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:01.648291 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.648253    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials" pod "e3afccc9-9458-461e-ab3c-88780f5b1ce9" (UID: "e3afccc9-9458-461e-ab3c-88780f5b1ce9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:01.747734 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.747660    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz8nj\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-kube-api-access-zz8nj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:01.747734 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:01.747689    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3afccc9-9458-461e-ab3c-88780f5b1ce9-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:02.048800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.048763    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"]
Nov 03 13:02:02.051727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.051680    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b55d695-nzcgg"]
Nov 03 13:02:02.522202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.522170    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-47tzn"]
Nov 03 13:02:02.522826 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.522810    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e3afccc9-9458-461e-ab3c-88780f5b1ce9" containerName="payload-processing"
Nov 03 13:02:02.522883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.522827    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3afccc9-9458-461e-ab3c-88780f5b1ce9" containerName="payload-processing"
Nov 03 13:02:02.522948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.522938    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3afccc9-9458-461e-ab3c-88780f5b1ce9" containerName="payload-processing"
Nov 03 13:02:02.527797 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.527777    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.576843 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.576818    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 13:02:02.577045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.576929    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 13:02:02.577045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.576935    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 13:02:02.577045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.576991    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 13:02:02.577045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.577015    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2d4a9f7ff9e0\""
Nov 03 13:02:02.579085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.579051    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-routing-overlay-volume\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.579284 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.579269    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hprjg\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-kube-api-access-hprjg\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.579354 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.579297    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-plugins-config-volume\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.579354 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.579338    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.622864 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.622834    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xn5pn\""
Nov 03 13:02:02.626087 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.626061    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-47tzn"]
Nov 03 13:02:02.680191 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.680159    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hprjg\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-kube-api-access-hprjg\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.680350 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.680221    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-plugins-config-volume\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.680350 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.680265    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.680350 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.680305    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-routing-overlay-volume\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.681212 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.681187    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-routing-overlay-volume\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.681388 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.681305    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-plugins-config-volume\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.683555 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.683536    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.733385 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.733354    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hprjg\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-kube-api-access-hprjg\") pod \"payload-processing-external-model-568cd79944-47tzn\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:02.851785 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:02.849952    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:03.123844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:03.123766    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3afccc9-9458-461e-ab3c-88780f5b1ce9" path="/var/lib/kubelet/pods/e3afccc9-9458-461e-ab3c-88780f5b1ce9/volumes"
Nov 03 13:02:03.333362 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:03.333324    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-47tzn"]
Nov 03 13:02:03.334276 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:02:03.334251    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6211a5bc_c90d_4334_922d_de73fc30b1ed.slice/crio-f44945b27fd9d328b71dc2826e2461fcc43309179ace6fd59573e1812ef1c00a WatchSource:0}: Error finding container f44945b27fd9d328b71dc2826e2461fcc43309179ace6fd59573e1812ef1c00a: Status 404 returned error can't find the container with id f44945b27fd9d328b71dc2826e2461fcc43309179ace6fd59573e1812ef1c00a
Nov 03 13:02:03.675785 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:03.674117    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn" event={"ID":"6211a5bc-c90d-4334-922d-de73fc30b1ed","Type":"ContainerStarted","Data":"9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075"}
Nov 03 13:02:03.692721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:03.680807    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn" event={"ID":"6211a5bc-c90d-4334-922d-de73fc30b1ed","Type":"ContainerStarted","Data":"f44945b27fd9d328b71dc2826e2461fcc43309179ace6fd59573e1812ef1c00a"}
Nov 03 13:02:03.692911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:03.692806    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:03.803501 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:03.803451    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn" podStartSLOduration=1.803436229 podStartE2EDuration="1.803436229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:02:03.801327521 +0000 UTC m=+4272.813794196" watchObservedRunningTime="2026-11-03 13:02:03.803436229 +0000 UTC m=+4272.815902898"
Nov 03 13:02:04.817919 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:04.817890    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d4a9f7ff9e0: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:04.817919 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:04.817913    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-568cd79944-47tzn: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:04.818502 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:04.817961    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials podName:6211a5bc-c90d-4334-922d-de73fc30b1ed nodeName:}" failed. No retries permitted until 2026-11-03 13:02:05.317946427 +0000 UTC m=+4274.330413067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials") pod "payload-processing-external-model-568cd79944-47tzn" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed") : secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:05.324216 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:05.324180    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d4a9f7ff9e0: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:05.324216 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:05.324208    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-568cd79944-47tzn: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:05.324428 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:05.324259    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials podName:6211a5bc-c90d-4334-922d-de73fc30b1ed nodeName:}" failed. No retries permitted until 2026-11-03 13:02:06.324245076 +0000 UTC m=+4275.336711721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials") pod "payload-processing-external-model-568cd79944-47tzn" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed") : secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:05.618795 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:05.618712    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-47tzn"]
Nov 03 13:02:06.357327 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:06.357295    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d4a9f7ff9e0: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:06.357327 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:06.357321    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-568cd79944-47tzn: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:06.357809 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:06.357369    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials podName:6211a5bc-c90d-4334-922d-de73fc30b1ed nodeName:}" failed. No retries permitted until 2026-11-03 13:02:08.357355014 +0000 UTC m=+4277.369821668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials") pod "payload-processing-external-model-568cd79944-47tzn" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed") : secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:08.395131 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:08.395098    4909 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d4a9f7ff9e0: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:08.395131 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:08.395125    4909 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-568cd79944-47tzn: secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:08.395595 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:08.395175    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials podName:6211a5bc-c90d-4334-922d-de73fc30b1ed nodeName:}" failed. No retries permitted until 2026-11-03 13:02:12.395159207 +0000 UTC m=+4281.407625848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials") pod "payload-processing-external-model-568cd79944-47tzn" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed") : secret "agc-cred-2d4a9f7ff9e0" not found
Nov 03 13:02:09.565299 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:09.565264    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:02:10.322115 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:10.322076    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"]
Nov 03 13:02:10.689781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:10.689720    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn" podUID="6211a5bc-c90d-4334-922d-de73fc30b1ed" containerName="payload-processing" containerID="cri-o://9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075" gracePeriod=25
Nov 03 13:02:10.989442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:10.989415    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:11.135874 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.135837    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-plugins-config-volume\") pod \"6211a5bc-c90d-4334-922d-de73fc30b1ed\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") "
Nov 03 13:02:11.136069 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.135933    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-routing-overlay-volume\") pod \"6211a5bc-c90d-4334-922d-de73fc30b1ed\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") "
Nov 03 13:02:11.136069 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.136051    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials\") pod \"6211a5bc-c90d-4334-922d-de73fc30b1ed\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") "
Nov 03 13:02:11.136210 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.136085    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-kube-api-access-hprjg\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-kube-api-access-hprjg\") pod \"6211a5bc-c90d-4334-922d-de73fc30b1ed\" (UID: \"6211a5bc-c90d-4334-922d-de73fc30b1ed\") "
Nov 03 13:02:11.136316 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.136288    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-plugins-config-volume" pod "6211a5bc-c90d-4334-922d-de73fc30b1ed" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:11.136387 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.136328    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-routing-overlay-volume" pod "6211a5bc-c90d-4334-922d-de73fc30b1ed" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:11.136586 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.136562    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:11.136679 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.136587    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6211a5bc-c90d-4334-922d-de73fc30b1ed-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:11.139047 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.139019    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials" pod "6211a5bc-c90d-4334-922d-de73fc30b1ed" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:11.139170 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.139154    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-kube-api-access-hprjg" pod "6211a5bc-c90d-4334-922d-de73fc30b1ed" (UID: "6211a5bc-c90d-4334-922d-de73fc30b1ed"). InnerVolumeSpecName "kube-api-access-hprjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:11.237607 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.237522    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:11.237607 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.237554    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hprjg\" (UniqueName: \"kubernetes.io/projected/6211a5bc-c90d-4334-922d-de73fc30b1ed-kube-api-access-hprjg\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:11.729324 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.729291    4909 generic.go:356] "Generic (PLEG): container finished" podID="6211a5bc-c90d-4334-922d-de73fc30b1ed" containerID="9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075" exitCode=0
Nov 03 13:02:11.729781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.729334    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn"
Nov 03 13:02:11.729781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.729373    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn" event={"ID":"6211a5bc-c90d-4334-922d-de73fc30b1ed","Type":"ContainerDied","Data":"9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075"}
Nov 03 13:02:11.729781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.729434    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-47tzn" event={"ID":"6211a5bc-c90d-4334-922d-de73fc30b1ed","Type":"ContainerDied","Data":"f44945b27fd9d328b71dc2826e2461fcc43309179ace6fd59573e1812ef1c00a"}
Nov 03 13:02:11.729781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.729465    4909 scope.go:122] "RemoveContainer" containerID="9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075"
Nov 03 13:02:11.740366 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.740348    4909 scope.go:122] "RemoveContainer" containerID="9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075"
Nov 03 13:02:11.740618 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:11.740599    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075\": container with ID starting with 9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075 not found: ID does not exist" containerID="9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075"
Nov 03 13:02:11.740662 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.740625    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075"} err="failed to get container status \"9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075\": rpc error: code = NotFound desc = could not find container \"9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075\": container with ID starting with 9901b2fc6bb6ff8ad837a5eaad3eb856c6c841e8045865c2a5cddf7263fc1075 not found: ID does not exist"
Nov 03 13:02:11.901672 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.901624    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-47tzn"]
Nov 03 13:02:11.946242 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:11.946211    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-47tzn"]
Nov 03 13:02:13.124834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:13.124803    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6211a5bc-c90d-4334-922d-de73fc30b1ed" path="/var/lib/kubelet/pods/6211a5bc-c90d-4334-922d-de73fc30b1ed/volumes"
Nov 03 13:02:14.162891 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.162858    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-blshd"]
Nov 03 13:02:14.163442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.163427    4909 cpu_manager.go:414] "containerMap: removing container" podUID="6211a5bc-c90d-4334-922d-de73fc30b1ed" containerName="payload-processing"
Nov 03 13:02:14.163492 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.163443    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6211a5bc-c90d-4334-922d-de73fc30b1ed" containerName="payload-processing"
Nov 03 13:02:14.163619 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.163606    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="6211a5bc-c90d-4334-922d-de73fc30b1ed" containerName="payload-processing"
Nov 03 13:02:14.168469 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.168452    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.220291 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.220268    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-blshd"]
Nov 03 13:02:14.262979 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.262955    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2d4a9f7ff9e0\""
Nov 03 13:02:14.262979 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.262967    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 13:02:14.263159 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.262980    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 13:02:14.263159 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.263001    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 13:02:14.263159 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.263021    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cthvn\""
Nov 03 13:02:14.263159 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.263042    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 13:02:14.278949 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.278928    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-plugins-config-volume\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.279051 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.278995    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-routing-overlay-volume\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.279113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.279093    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-provider-credentials\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.279165 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.279149    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz67r\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-kube-api-access-qz67r\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.380645 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.380610    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-routing-overlay-volume\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.380819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.380691    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-provider-credentials\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.380819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.380785    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz67r\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-kube-api-access-qz67r\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.380951 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.380856    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-plugins-config-volume\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.381446 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.381413    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-routing-overlay-volume\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.381565 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.381540    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-plugins-config-volume\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.383191 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.383169    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-provider-credentials\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.433126 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.433066    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz67r\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-kube-api-access-qz67r\") pod \"payload-processing-external-model-568cd79944-blshd\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") " pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.482268 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.482239    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:14.853217 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:14.853181    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-blshd"]
Nov 03 13:02:14.854980 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:02:14.854945    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1227f7_f777_4871_a004_103e8c24fff4.slice/crio-d188d3c130df04cec6c92ec6997b78a8a82d64b40dcb114347de40d9964f6b19 WatchSource:0}: Error finding container d188d3c130df04cec6c92ec6997b78a8a82d64b40dcb114347de40d9964f6b19: Status 404 returned error can't find the container with id d188d3c130df04cec6c92ec6997b78a8a82d64b40dcb114347de40d9964f6b19
Nov 03 13:02:15.322503 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.322466    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q" podUID="8d3b4dd9-b21e-4f1b-bc82-517349e20f12" containerName="payload-pre-processing" containerID="cri-o://14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c" gracePeriod=25
Nov 03 13:02:15.592876 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.592817    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 13:02:15.768457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.768428    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-plugins-config-volume\") pod \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") "
Nov 03 13:02:15.768656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.768624    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-kube-api-access-m65d7\" (UniqueName: \"kubernetes.io/projected/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-kube-api-access-m65d7\") pod \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\" (UID: \"8d3b4dd9-b21e-4f1b-bc82-517349e20f12\") "
Nov 03 13:02:15.768829 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.768808    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-plugins-config-volume" pod "8d3b4dd9-b21e-4f1b-bc82-517349e20f12" (UID: "8d3b4dd9-b21e-4f1b-bc82-517349e20f12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:15.768884 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.768851    4909 generic.go:356] "Generic (PLEG): container finished" podID="8d3b4dd9-b21e-4f1b-bc82-517349e20f12" containerID="14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c" exitCode=0
Nov 03 13:02:15.769003 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.768987    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q" event={"ID":"8d3b4dd9-b21e-4f1b-bc82-517349e20f12","Type":"ContainerDied","Data":"14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c"}
Nov 03 13:02:15.769049 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.769031    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q" event={"ID":"8d3b4dd9-b21e-4f1b-bc82-517349e20f12","Type":"ContainerDied","Data":"f7cfbd5dc62497ab77cf479db446100ee0ddb60e43ec51e24e87d17e79bc3bd3"}
Nov 03 13:02:15.769086 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.769055    4909 scope.go:122] "RemoveContainer" containerID="14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c"
Nov 03 13:02:15.769086 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.769073    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:15.769184 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.769150    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"
Nov 03 13:02:15.771002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.770979    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd" event={"ID":"6f1227f7-f777-4871-a004-103e8c24fff4","Type":"ContainerStarted","Data":"cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11"}
Nov 03 13:02:15.771143 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.771028    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd" event={"ID":"6f1227f7-f777-4871-a004-103e8c24fff4","Type":"ContainerStarted","Data":"d188d3c130df04cec6c92ec6997b78a8a82d64b40dcb114347de40d9964f6b19"}
Nov 03 13:02:15.771143 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.771085    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:15.771251 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.771215    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-kube-api-access-m65d7" pod "8d3b4dd9-b21e-4f1b-bc82-517349e20f12" (UID: "8d3b4dd9-b21e-4f1b-bc82-517349e20f12"). InnerVolumeSpecName "kube-api-access-m65d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:15.787318 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.787300    4909 scope.go:122] "RemoveContainer" containerID="14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c"
Nov 03 13:02:15.787616 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:15.787591    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c\": container with ID starting with 14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c not found: ID does not exist" containerID="14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c"
Nov 03 13:02:15.787669 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.787625    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c"} err="failed to get container status \"14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c\": rpc error: code = NotFound desc = could not find container \"14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c\": container with ID starting with 14c7fa1e963b5e9655b97f92e317d2f555cc19e2817fc05322efbe7bed136d0c not found: ID does not exist"
Nov 03 13:02:15.870016 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.869933    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65d7\" (UniqueName: \"kubernetes.io/projected/8d3b4dd9-b21e-4f1b-bc82-517349e20f12-kube-api-access-m65d7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:15.927851 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:15.927799    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd" podStartSLOduration=1.9277835859999999 podStartE2EDuration="1.927783586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:02:15.925869448 +0000 UTC m=+4284.938336112" watchObservedRunningTime="2026-11-03 13:02:15.927783586 +0000 UTC m=+4284.940250248"
Nov 03 13:02:16.191346 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:16.191310    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"]
Nov 03 13:02:16.195011 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:16.194988    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aa9221-6bd7d989b7-g4x5q"]
Nov 03 13:02:17.124155 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:17.124122    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3b4dd9-b21e-4f1b-bc82-517349e20f12" path="/var/lib/kubelet/pods/8d3b4dd9-b21e-4f1b-bc82-517349e20f12/volumes"
Nov 03 13:02:17.535140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:17.535097    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-blshd"]
Nov 03 13:02:22.791521 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:22.791466    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd" podUID="6f1227f7-f777-4871-a004-103e8c24fff4" containerName="payload-processing" containerID="cri-o://cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11" gracePeriod=25
Nov 03 13:02:23.068450 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.068421    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:23.165960 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.165929    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-kube-api-access-qz67r\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-kube-api-access-qz67r\") pod \"6f1227f7-f777-4871-a004-103e8c24fff4\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") "
Nov 03 13:02:23.166144 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.165988    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-provider-credentials\") pod \"6f1227f7-f777-4871-a004-103e8c24fff4\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") "
Nov 03 13:02:23.166144 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.166074    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-plugins-config-volume\") pod \"6f1227f7-f777-4871-a004-103e8c24fff4\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") "
Nov 03 13:02:23.166238 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.166169    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-routing-overlay-volume\") pod \"6f1227f7-f777-4871-a004-103e8c24fff4\" (UID: \"6f1227f7-f777-4871-a004-103e8c24fff4\") "
Nov 03 13:02:23.166786 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.166510    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-plugins-config-volume" pod "6f1227f7-f777-4871-a004-103e8c24fff4" (UID: "6f1227f7-f777-4871-a004-103e8c24fff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:23.166786 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.166518    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-routing-overlay-volume" pod "6f1227f7-f777-4871-a004-103e8c24fff4" (UID: "6f1227f7-f777-4871-a004-103e8c24fff4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:23.166786 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.166736    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:23.166786 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.166757    4909 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f1227f7-f777-4871-a004-103e8c24fff4-routing-overlay-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:23.168149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.168126    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-kube-api-access-qz67r" pod "6f1227f7-f777-4871-a004-103e8c24fff4" (UID: "6f1227f7-f777-4871-a004-103e8c24fff4"). InnerVolumeSpecName "kube-api-access-qz67r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:23.168265 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.168213    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-provider-credentials" pod "6f1227f7-f777-4871-a004-103e8c24fff4" (UID: "6f1227f7-f777-4871-a004-103e8c24fff4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:23.268093 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.268063    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz67r\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-kube-api-access-qz67r\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:23.268093 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.268089    4909 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f1227f7-f777-4871-a004-103e8c24fff4-provider-credentials\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:23.835498 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.835464    4909 generic.go:356] "Generic (PLEG): container finished" podID="6f1227f7-f777-4871-a004-103e8c24fff4" containerID="cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11" exitCode=0
Nov 03 13:02:23.835948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.835528    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd" event={"ID":"6f1227f7-f777-4871-a004-103e8c24fff4","Type":"ContainerDied","Data":"cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11"}
Nov 03 13:02:23.835948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.835600    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd" event={"ID":"6f1227f7-f777-4871-a004-103e8c24fff4","Type":"ContainerDied","Data":"d188d3c130df04cec6c92ec6997b78a8a82d64b40dcb114347de40d9964f6b19"}
Nov 03 13:02:23.835948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.835532    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-568cd79944-blshd"
Nov 03 13:02:23.835948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.835631    4909 scope.go:122] "RemoveContainer" containerID="cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11"
Nov 03 13:02:23.845961 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.845938    4909 scope.go:122] "RemoveContainer" containerID="cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11"
Nov 03 13:02:23.846222 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:23.846200    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11\": container with ID starting with cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11 not found: ID does not exist" containerID="cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11"
Nov 03 13:02:23.846281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.846230    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11"} err="failed to get container status \"cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11\": rpc error: code = NotFound desc = could not find container \"cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11\": container with ID starting with cbed773e7d75433854765105b052aa7b68728500769a426c7289a2ae8cbf6f11 not found: ID does not exist"
Nov 03 13:02:23.988114 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:23.988082    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-blshd"]
Nov 03 13:02:24.034057 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:24.034030    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-568cd79944-blshd"]
Nov 03 13:02:25.124340 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:25.124291    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1227f7-f777-4871-a004-103e8c24fff4" path="/var/lib/kubelet/pods/6f1227f7-f777-4871-a004-103e8c24fff4/volumes"
Nov 03 13:02:27.731086 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.731053    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"]
Nov 03 13:02:27.731777 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.731760    4909 cpu_manager.go:414] "containerMap: removing container" podUID="6f1227f7-f777-4871-a004-103e8c24fff4" containerName="payload-processing"
Nov 03 13:02:27.731850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.731778    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1227f7-f777-4871-a004-103e8c24fff4" containerName="payload-processing"
Nov 03 13:02:27.731850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.731794    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8d3b4dd9-b21e-4f1b-bc82-517349e20f12" containerName="payload-pre-processing"
Nov 03 13:02:27.731850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.731800    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3b4dd9-b21e-4f1b-bc82-517349e20f12" containerName="payload-pre-processing"
Nov 03 13:02:27.731956 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.731916    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1227f7-f777-4871-a004-103e8c24fff4" containerName="payload-processing"
Nov 03 13:02:27.731956 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.731929    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3b4dd9-b21e-4f1b-bc82-517349e20f12" containerName="payload-pre-processing"
Nov 03 13:02:27.736584 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.736561    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:27.737447 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.737419    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfcr\" (UniqueName: \"kubernetes.io/projected/0d79b642-d783-48ad-bde9-ae2867ec77c0-kube-api-access-7bfcr\") pod \"payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:27.737555 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.737541    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d79b642-d783-48ad-bde9-ae2867ec77c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:27.785522 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.785492    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6bd346ef\""
Nov 03 13:02:27.785640 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.785596    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6bd346ef-dockercfg-bq86l\""
Nov 03 13:02:27.790156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.790128    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"]
Nov 03 13:02:27.839304 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.839260    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d79b642-d783-48ad-bde9-ae2867ec77c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:27.839518 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.839478    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfcr\" (UniqueName: \"kubernetes.io/projected/0d79b642-d783-48ad-bde9-ae2867ec77c0-kube-api-access-7bfcr\") pod \"payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:27.840660 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.840630    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d79b642-d783-48ad-bde9-ae2867ec77c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:27.911948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.911904    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"]
Nov 03 13:02:27.923201 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.922764    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:27.934512 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.934470    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfcr\" (UniqueName: \"kubernetes.io/projected/0d79b642-d783-48ad-bde9-ae2867ec77c0-kube-api-access-7bfcr\") pod \"payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:27.994831 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:27.994762    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"]
Nov 03 13:02:28.043223 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.043192    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdxq\" (UniqueName: \"kubernetes.io/projected/e9410a32-a6c4-4ac3-83aa-8c263865750c-kube-api-access-9kdxq\") pod \"payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") " pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.043397 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.043234    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9410a32-a6c4-4ac3-83aa-8c263865750c-plugins-config-volume\") pod \"payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") " pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.050777 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.050754    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:28.144253 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.144223    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdxq\" (UniqueName: \"kubernetes.io/projected/e9410a32-a6c4-4ac3-83aa-8c263865750c-kube-api-access-9kdxq\") pod \"payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") " pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.144390 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.144272    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9410a32-a6c4-4ac3-83aa-8c263865750c-plugins-config-volume\") pod \"payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") " pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.145049 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.145029    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9410a32-a6c4-4ac3-83aa-8c263865750c-plugins-config-volume\") pod \"payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") " pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.198004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.197970    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdxq\" (UniqueName: \"kubernetes.io/projected/e9410a32-a6c4-4ac3-83aa-8c263865750c-kube-api-access-9kdxq\") pod \"payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") " pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.264770 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.264670    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.419756 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.419724    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"]
Nov 03 13:02:28.421411 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:02:28.421386    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d79b642_d783_48ad_bde9_ae2867ec77c0.slice/crio-f6fbae74878afd530e0caf7c1d371ecc85603733ce7abfdda13e16bc40414e7b WatchSource:0}: Error finding container f6fbae74878afd530e0caf7c1d371ecc85603733ce7abfdda13e16bc40414e7b: Status 404 returned error can't find the container with id f6fbae74878afd530e0caf7c1d371ecc85603733ce7abfdda13e16bc40414e7b
Nov 03 13:02:28.635222 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.634182    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"]
Nov 03 13:02:28.638080 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:02:28.638051    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9410a32_a6c4_4ac3_83aa_8c263865750c.slice/crio-71f31fb107009f322a315a3d6af9a8c41aa28dea598e319fea1fe865bfac24d9 WatchSource:0}: Error finding container 71f31fb107009f322a315a3d6af9a8c41aa28dea598e319fea1fe865bfac24d9: Status 404 returned error can't find the container with id 71f31fb107009f322a315a3d6af9a8c41aa28dea598e319fea1fe865bfac24d9
Nov 03 13:02:28.886924 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.886822    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn" event={"ID":"0d79b642-d783-48ad-bde9-ae2867ec77c0","Type":"ContainerStarted","Data":"c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc"}
Nov 03 13:02:28.886924 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.886882    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn" event={"ID":"0d79b642-d783-48ad-bde9-ae2867ec77c0","Type":"ContainerStarted","Data":"f6fbae74878afd530e0caf7c1d371ecc85603733ce7abfdda13e16bc40414e7b"}
Nov 03 13:02:28.887444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.887044    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:28.888395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.888371    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz" event={"ID":"e9410a32-a6c4-4ac3-83aa-8c263865750c","Type":"ContainerStarted","Data":"15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de"}
Nov 03 13:02:28.888523 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.888407    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz" event={"ID":"e9410a32-a6c4-4ac3-83aa-8c263865750c","Type":"ContainerStarted","Data":"71f31fb107009f322a315a3d6af9a8c41aa28dea598e319fea1fe865bfac24d9"}
Nov 03 13:02:28.888523 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.888438    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:28.996398 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:28.996352    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn" podStartSLOduration=1.996339485 podStartE2EDuration="1.996339485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:02:28.993468749 +0000 UTC m=+4298.005935423" watchObservedRunningTime="2026-11-03 13:02:28.996339485 +0000 UTC m=+4298.008806147"
Nov 03 13:02:33.999139 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:33.994489    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz" podStartSLOduration=6.994472953 podStartE2EDuration="6.994472953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:02:29.098770183 +0000 UTC m=+4298.111236847" watchObservedRunningTime="2026-11-03 13:02:33.994472953 +0000 UTC m=+4303.006939616"
Nov 03 13:02:33.999139 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:33.996018    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:02:39.565788 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:39.565748    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:02:39.898622 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:39.898535    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:02:39.898622 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:39.898610    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:02:56.759315 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.759280    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"]
Nov 03 13:02:56.765383 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.765359    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.831236 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.831199    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-metrics-certs\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.831403 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.831357    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.831403 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.831383    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nzp\" (UniqueName: \"kubernetes.io/projected/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-kube-api-access-n7nzp\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.831554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.831518    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.861478 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.861450    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d4609f-serving-cert\""
Nov 03 13:02:56.865064 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.865044    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"]
Nov 03 13:02:56.933035 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.933008    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.933183 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.933102    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-metrics-certs\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.933183 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:56.933166    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-serving-cert: secret "maas-api-e2e-inf-a-d4609f-serving-cert" not found
Nov 03 13:02:56.933268 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:02:56.933228    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls podName:96ca5fbc-b2cc-4e72-b046-9c94854cf60a nodeName:}" failed. No retries permitted until 2026-11-03 13:02:57.433210451 +0000 UTC m=+4326.445677104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls") pod "maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz" (UID: "96ca5fbc-b2cc-4e72-b046-9c94854cf60a") : secret "maas-api-e2e-inf-a-d4609f-serving-cert" not found
Nov 03 13:02:56.933323 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.933289    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.933323 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.933318    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nzp\" (UniqueName: \"kubernetes.io/projected/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-kube-api-access-n7nzp\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.933989 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.933965    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.935572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.935552    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-metrics-certs\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:56.988878 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:56.988856    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nzp\" (UniqueName: \"kubernetes.io/projected/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-kube-api-access-n7nzp\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:57.439220 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:57.439184    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:57.441615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:57.441588    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:57.679120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:57.679081    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:58.054058 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:58.054027    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"]
Nov 03 13:02:58.054468 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:02:58.054439    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ca5fbc_b2cc_4e72_b046_9c94854cf60a.slice/crio-a7456b8105098e2aa40ad1d49f3c927d63ab1da6c3094d98f9a3caf6f5f61da4 WatchSource:0}: Error finding container a7456b8105098e2aa40ad1d49f3c927d63ab1da6c3094d98f9a3caf6f5f61da4: Status 404 returned error can't find the container with id a7456b8105098e2aa40ad1d49f3c927d63ab1da6c3094d98f9a3caf6f5f61da4
Nov 03 13:02:58.055829 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:58.055812    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:02:58.133556 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:58.133528    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz" event={"ID":"96ca5fbc-b2cc-4e72-b046-9c94854cf60a","Type":"ContainerStarted","Data":"a7456b8105098e2aa40ad1d49f3c927d63ab1da6c3094d98f9a3caf6f5f61da4"}
Nov 03 13:02:59.141670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:59.141631    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz" event={"ID":"96ca5fbc-b2cc-4e72-b046-9c94854cf60a","Type":"ContainerStarted","Data":"046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43"}
Nov 03 13:02:59.142103 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:59.141793    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:02:59.285929 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:02:59.285841    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz" podStartSLOduration=2.326366604 podStartE2EDuration="3.285828745s" totalImagesPullingTime="959.462141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:02:59.285333636 +0000 UTC m=+4328.297800300" watchObservedRunningTime="2026-11-03 13:02:59.285828745 +0000 UTC m=+4328.298295408"
Nov 03 13:03:05.154534 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:05.154505    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:03:09.564129 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:03:09.564092    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:03:27.328732 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:27.328687    4909 generic.go:356] "Generic (PLEG): container finished" podID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerID="8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43" exitCode=0
Nov 03 13:03:27.329213 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:27.328759    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" event={"ID":"283a09fd-6d12-43f2-8672-815f5f4d7863","Type":"ContainerDied","Data":"8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43"}
Nov 03 13:03:27.329213 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:27.329115    4909 scope.go:122] "RemoveContainer" containerID="8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43"
Nov 03 13:03:27.330686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:27.330657    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:03:27.330777 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:27.330740    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:03:27.330819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:27.330774    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:03:28.371625 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:28.371587    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" event={"ID":"283a09fd-6d12-43f2-8672-815f5f4d7863","Type":"ContainerStarted","Data":"6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05"}
Nov 03 13:03:29.130492 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:29.130459    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:03:29.131843 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:29.131814    4909 patch_prober.go:29] interesting pod/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:03:29.131976 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:29.131867    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 13:03:30.131000 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:30.130968    4909 patch_prober.go:29] interesting pod/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:03:30.131474 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:30.131022    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 13:03:31.130737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:31.130688    4909 patch_prober.go:29] interesting pod/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:03:31.130910 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:31.130754    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 13:03:32.130791 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:32.130753    4909 patch_prober.go:29] interesting pod/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:03:32.131271 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:32.130808    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 13:03:33.131348 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:33.131308    4909 patch_prober.go:29] interesting pod/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:03:33.131763 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:33.131362    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 13:03:33.379835 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:33.379798    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:03:34.135939 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:34.135908    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:03:34.137251 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:34.137227    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:03:34.790663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:34.790628    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:03:39.563956 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:03:39.563915    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:03:42.869202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.869168    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"]
Nov 03 13:03:42.873457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.873436    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:42.924458 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.924437    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6bd346ef-serving-cert\""
Nov 03 13:03:42.929007 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.928985    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"]
Nov 03 13:03:42.985489 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.985464    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:42.985636 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.985522    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6868f82-799e-4b22-acac-036fc39682c5-openshift-service-ca\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:42.985723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.985676    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-metrics-certs\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:42.985784 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:42.985748    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhkl2\" (UniqueName: \"kubernetes.io/projected/a6868f82-799e-4b22-acac-036fc39682c5-kube-api-access-nhkl2\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.086599 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.086568    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-metrics-certs\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.086775 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.086623    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhkl2\" (UniqueName: \"kubernetes.io/projected/a6868f82-799e-4b22-acac-036fc39682c5-kube-api-access-nhkl2\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.086775 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.086760    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.086912 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.086826    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6868f82-799e-4b22-acac-036fc39682c5-openshift-service-ca\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.086912 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:03:43.086892    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-serving-cert: secret "maas-api-e2e-mt-6bd346ef-serving-cert" not found
Nov 03 13:03:43.087012 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:03:43.086978    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls podName:a6868f82-799e-4b22-acac-036fc39682c5 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:43.586957062 +0000 UTC m=+4372.599423712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls") pod "maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8" (UID: "a6868f82-799e-4b22-acac-036fc39682c5") : secret "maas-api-e2e-mt-6bd346ef-serving-cert" not found
Nov 03 13:03:43.087465 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.087441    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6868f82-799e-4b22-acac-036fc39682c5-openshift-service-ca\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.089075 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.089050    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-metrics-certs\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.161349 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.161250    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhkl2\" (UniqueName: \"kubernetes.io/projected/a6868f82-799e-4b22-acac-036fc39682c5-kube-api-access-nhkl2\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.593573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.593541    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.595999 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.595978    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls\") pod \"maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:43.787998 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:43.787962    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:44.156039 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:44.156011    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"]
Nov 03 13:03:44.157207 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:03:44.157170    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6868f82_799e_4b22_acac_036fc39682c5.slice/crio-45f1a9d789dd4a51725f19b4d2bb86097cad12d68a84da2963987235173efb3a WatchSource:0}: Error finding container 45f1a9d789dd4a51725f19b4d2bb86097cad12d68a84da2963987235173efb3a: Status 404 returned error can't find the container with id 45f1a9d789dd4a51725f19b4d2bb86097cad12d68a84da2963987235173efb3a
Nov 03 13:03:44.485181 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:44.485145    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8" event={"ID":"a6868f82-799e-4b22-acac-036fc39682c5","Type":"ContainerStarted","Data":"45f1a9d789dd4a51725f19b4d2bb86097cad12d68a84da2963987235173efb3a"}
Nov 03 13:03:45.492912 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:45.492875    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8" event={"ID":"a6868f82-799e-4b22-acac-036fc39682c5","Type":"ContainerStarted","Data":"dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962"}
Nov 03 13:03:45.493357 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:45.493006    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:03:45.605426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:45.605296    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8" podStartSLOduration=2.658589492 podStartE2EDuration="3.605278117s" totalImagesPullingTime="946.688625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:45.604012242 +0000 UTC m=+4374.616478904" watchObservedRunningTime="2026-11-03 13:03:45.605278117 +0000 UTC m=+4374.617744781"
Nov 03 13:03:51.504830 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:03:51.504797    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:04:09.564262 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:04:09.564228    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:04:28.376818 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:04:28.376784    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:04:39.565174 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:04:39.565133    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:04:52.795663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:04:52.795629    4909 scope.go:122] "RemoveContainer" containerID="f1f7e3e2158dbbc1a756c5eeb004d311471ad1039f1456c1a1363a5ee357f038"
Nov 03 13:05:09.564570 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:05:09.564490    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:05:21.639568 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.639534    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"]
Nov 03 13:05:21.644208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.644187    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.685649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.685617    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-home\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.685801 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.685718    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tls-certs\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.685801 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.685740    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rzbb\" (UniqueName: \"kubernetes.io/projected/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-kube-api-access-5rzbb\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.685801 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.685793    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tmp-dir\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.685971 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.685900    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-model-cache\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.685971 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.685928    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-dshm\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.740180 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.740153    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"kube-root-ca.crt\""
Nov 03 13:05:21.740180 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.740167    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"rate-test-model-1250b3-kserve-self-signed-certs\""
Nov 03 13:05:21.740180 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.740171    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"openshift-service-ca.crt\""
Nov 03 13:05:21.740398 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.740154    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1250b3\"/\"default-dockercfg-rp9tp\""
Nov 03 13:05:21.743555 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.743535    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"]
Nov 03 13:05:21.829649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.822623    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-model-cache\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.829649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.822683    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-dshm\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.829649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.822762    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-home\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.829649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.822809    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tls-certs\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.829649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.822834    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rzbb\" (UniqueName: \"kubernetes.io/projected/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-kube-api-access-5rzbb\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.829649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.822923    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tmp-dir\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.829649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.823482    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tmp-dir\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.830333 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.830307    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-model-cache\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.830553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.830530    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-home\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.834096 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.833848    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-dshm\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.834096 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.834036    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tls-certs\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.883438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.883411    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rzbb\" (UniqueName: \"kubernetes.io/projected/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-kube-api-access-5rzbb\") pod \"rate-test-model-1250b3-kserve-5d8f5d7d65-j9845\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") " pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:21.956365 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:21.956334    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:22.324675 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:22.324644    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"]
Nov 03 13:05:22.325807 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:05:22.325770    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3b0c8b9_80ef_45dc_ba0f_8852a040dee6.slice/crio-4ae8d228a58f6d66d2e0b94a8bbde31be6e4de2cc6fc04b319ac4f17c6f34730 WatchSource:0}: Error finding container 4ae8d228a58f6d66d2e0b94a8bbde31be6e4de2cc6fc04b319ac4f17c6f34730: Status 404 returned error can't find the container with id 4ae8d228a58f6d66d2e0b94a8bbde31be6e4de2cc6fc04b319ac4f17c6f34730
Nov 03 13:05:23.207322 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:23.207279    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845" event={"ID":"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6","Type":"ContainerStarted","Data":"fdbc6e69e7832089b8973acf7f088e9a307edffd9b854e83b00f4464ef474f06"}
Nov 03 13:05:23.207748 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:23.207339    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845" event={"ID":"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6","Type":"ContainerStarted","Data":"4ae8d228a58f6d66d2e0b94a8bbde31be6e4de2cc6fc04b319ac4f17c6f34730"}
Nov 03 13:05:23.316033 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:23.315977    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845" podStartSLOduration=2.315963103 podStartE2EDuration="2.315963103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:23.312464745 +0000 UTC m=+4472.324931409" watchObservedRunningTime="2026-11-03 13:05:23.315963103 +0000 UTC m=+4472.328429765"
Nov 03 13:05:23.956957 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:23.956845    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:31.957221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:31.957181    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:31.970067 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:31.970038    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:32.288457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:32.288383    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:05:37.000212 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:37.000176    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:05:39.564872 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:05:39.564833    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:05:51.725191 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:51.725165    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:05:51.735426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:05:51.735404    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:06:09.564543 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:06:09.564504    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:06:15.349615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:15.349578    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:06:38.998901 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:38.998784    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zjqz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:06:38.999545 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:38.999494    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:06:38.999887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:38.999855    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:06:39.000005 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:38.999921    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:06:39.565340 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:06:39.565303    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:06:51.299867 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:51.298977    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"]
Nov 03 13:06:51.378523 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:51.378493    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"]
Nov 03 13:06:56.300192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.300148    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz" podUID="e9410a32-a6c4-4ac3-83aa-8c263865750c" containerName="payload-processing" containerID="cri-o://15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de" gracePeriod=25
Nov 03 13:06:56.379323 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.379265    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn" podUID="0d79b642-d783-48ad-bde9-ae2867ec77c0" containerName="payload-pre-processing" containerID="cri-o://c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc" gracePeriod=25
Nov 03 13:06:56.653613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.653582    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:06:56.675661 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.675629    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9410a32-a6c4-4ac3-83aa-8c263865750c-kube-api-access-9kdxq\" (UniqueName: \"kubernetes.io/projected/e9410a32-a6c4-4ac3-83aa-8c263865750c-kube-api-access-9kdxq\") pod \"e9410a32-a6c4-4ac3-83aa-8c263865750c\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") "
Nov 03 13:06:56.675812 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.675733    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9410a32-a6c4-4ac3-83aa-8c263865750c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9410a32-a6c4-4ac3-83aa-8c263865750c-plugins-config-volume\") pod \"e9410a32-a6c4-4ac3-83aa-8c263865750c\" (UID: \"e9410a32-a6c4-4ac3-83aa-8c263865750c\") "
Nov 03 13:06:56.676150 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.676119    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9410a32-a6c4-4ac3-83aa-8c263865750c-plugins-config-volume" pod "e9410a32-a6c4-4ac3-83aa-8c263865750c" (UID: "e9410a32-a6c4-4ac3-83aa-8c263865750c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:06:56.677929 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.677905    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9410a32-a6c4-4ac3-83aa-8c263865750c-kube-api-access-9kdxq" pod "e9410a32-a6c4-4ac3-83aa-8c263865750c" (UID: "e9410a32-a6c4-4ac3-83aa-8c263865750c"). InnerVolumeSpecName "kube-api-access-9kdxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:06:56.699792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.699761    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:06:56.776974 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.776944    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d79b642-d783-48ad-bde9-ae2867ec77c0-kube-api-access-7bfcr\" (UniqueName: \"kubernetes.io/projected/0d79b642-d783-48ad-bde9-ae2867ec77c0-kube-api-access-7bfcr\") pod \"0d79b642-d783-48ad-bde9-ae2867ec77c0\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") "
Nov 03 13:06:56.777157 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.776984    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d79b642-d783-48ad-bde9-ae2867ec77c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d79b642-d783-48ad-bde9-ae2867ec77c0-plugins-config-volume\") pod \"0d79b642-d783-48ad-bde9-ae2867ec77c0\" (UID: \"0d79b642-d783-48ad-bde9-ae2867ec77c0\") "
Nov 03 13:06:56.777447 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.777417    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdxq\" (UniqueName: \"kubernetes.io/projected/e9410a32-a6c4-4ac3-83aa-8c263865750c-kube-api-access-9kdxq\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:06:56.777447 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.777444    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9410a32-a6c4-4ac3-83aa-8c263865750c-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:06:56.777603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.777447    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d79b642-d783-48ad-bde9-ae2867ec77c0-plugins-config-volume" pod "0d79b642-d783-48ad-bde9-ae2867ec77c0" (UID: "0d79b642-d783-48ad-bde9-ae2867ec77c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:06:56.779421 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.779391    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d79b642-d783-48ad-bde9-ae2867ec77c0-kube-api-access-7bfcr" pod "0d79b642-d783-48ad-bde9-ae2867ec77c0" (UID: "0d79b642-d783-48ad-bde9-ae2867ec77c0"). InnerVolumeSpecName "kube-api-access-7bfcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:06:56.878472 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.878405    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfcr\" (UniqueName: \"kubernetes.io/projected/0d79b642-d783-48ad-bde9-ae2867ec77c0-kube-api-access-7bfcr\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:06:56.878472 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.878433    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d79b642-d783-48ad-bde9-ae2867ec77c0-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:06:56.906994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.906966    4909 generic.go:356] "Generic (PLEG): container finished" podID="0d79b642-d783-48ad-bde9-ae2867ec77c0" containerID="c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc" exitCode=0
Nov 03 13:06:56.907147 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.907034    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"
Nov 03 13:06:56.907147 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.907063    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn" event={"ID":"0d79b642-d783-48ad-bde9-ae2867ec77c0","Type":"ContainerDied","Data":"c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc"}
Nov 03 13:06:56.907147 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.907115    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn" event={"ID":"0d79b642-d783-48ad-bde9-ae2867ec77c0","Type":"ContainerDied","Data":"f6fbae74878afd530e0caf7c1d371ecc85603733ce7abfdda13e16bc40414e7b"}
Nov 03 13:06:56.907147 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.907143    4909 scope.go:122] "RemoveContainer" containerID="c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc"
Nov 03 13:06:56.908922 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.908898    4909 generic.go:356] "Generic (PLEG): container finished" podID="e9410a32-a6c4-4ac3-83aa-8c263865750c" containerID="15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de" exitCode=0
Nov 03 13:06:56.909082 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.909040    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz" event={"ID":"e9410a32-a6c4-4ac3-83aa-8c263865750c","Type":"ContainerDied","Data":"15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de"}
Nov 03 13:06:56.909082 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.909073    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz" event={"ID":"e9410a32-a6c4-4ac3-83aa-8c263865750c","Type":"ContainerDied","Data":"71f31fb107009f322a315a3d6af9a8c41aa28dea598e319fea1fe865bfac24d9"}
Nov 03 13:06:56.909234 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.909151    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"
Nov 03 13:06:56.918606 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.918510    4909 scope.go:122] "RemoveContainer" containerID="c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc"
Nov 03 13:06:56.918842 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:06:56.918815    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc\": container with ID starting with c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc not found: ID does not exist" containerID="c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc"
Nov 03 13:06:56.918912 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.918853    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc"} err="failed to get container status \"c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc\": rpc error: code = NotFound desc = could not find container \"c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc\": container with ID starting with c1f2a73c72c27002f18347e7383f5d7be38290522a33c268f0af38f2b0f9e1cc not found: ID does not exist"
Nov 03 13:06:56.918912 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.918878    4909 scope.go:122] "RemoveContainer" containerID="15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de"
Nov 03 13:06:56.929091 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.929075    4909 scope.go:122] "RemoveContainer" containerID="15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de"
Nov 03 13:06:56.929328 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:06:56.929309    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de\": container with ID starting with 15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de not found: ID does not exist" containerID="15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de"
Nov 03 13:06:56.929386 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:56.929337    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de"} err="failed to get container status \"15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de\": rpc error: code = NotFound desc = could not find container \"15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de\": container with ID starting with 15158fff9490a98d49d4dbeee41f4205c46a717dceeb388b7c92bf87778557de not found: ID does not exist"
Nov 03 13:06:57.059032 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:57.059005    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"]
Nov 03 13:06:57.062041 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:57.062022    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bd346ef-6d8bd55647-lrplz"]
Nov 03 13:06:57.124066 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:57.124042    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9410a32-a6c4-4ac3-83aa-8c263865750c" path="/var/lib/kubelet/pods/e9410a32-a6c4-4ac3-83aa-8c263865750c/volumes"
Nov 03 13:06:57.162759 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:57.162706    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"]
Nov 03 13:06:57.168352 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:57.168323    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6bd346ef-679dc49f95-kvmtn"]
Nov 03 13:06:59.123787 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:06:59.123757    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d79b642-d783-48ad-bde9-ae2867ec77c0" path="/var/lib/kubelet/pods/0d79b642-d783-48ad-bde9-ae2867ec77c0/volumes"
Nov 03 13:07:09.564133 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:07:09.564096    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:07:11.144371 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:11.144339    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:07:31.099919 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.099886    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"]
Nov 03 13:07:31.100500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.100485    4909 cpu_manager.go:414] "containerMap: removing container" podUID="0d79b642-d783-48ad-bde9-ae2867ec77c0" containerName="payload-pre-processing"
Nov 03 13:07:31.100553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.100500    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d79b642-d783-48ad-bde9-ae2867ec77c0" containerName="payload-pre-processing"
Nov 03 13:07:31.100553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.100544    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e9410a32-a6c4-4ac3-83aa-8c263865750c" containerName="payload-processing"
Nov 03 13:07:31.100553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.100549    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9410a32-a6c4-4ac3-83aa-8c263865750c" containerName="payload-processing"
Nov 03 13:07:31.100687 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.100674    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d79b642-d783-48ad-bde9-ae2867ec77c0" containerName="payload-pre-processing"
Nov 03 13:07:31.100759 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.100717    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9410a32-a6c4-4ac3-83aa-8c263865750c" containerName="payload-processing"
Nov 03 13:07:31.104219 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.104201    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.199649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.199609    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d4609f\"/\"openshift-service-ca.crt\""
Nov 03 13:07:31.199649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.199641    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d4609f\"/\"kube-root-ca.crt\""
Nov 03 13:07:31.199873 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.199687    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d4609f\"/\"test-model-d4609f-kserve-self-signed-certs\""
Nov 03 13:07:31.199873 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.199759    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d4609f\"/\"default-dockercfg-k424g\""
Nov 03 13:07:31.202909 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.202890    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"]
Nov 03 13:07:31.206398 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.206380    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tls-certs\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.206453 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.206416    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-home\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.206533 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.206515    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-model-cache\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.206667 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.206653    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-dshm\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.206714 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.206677    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzkbq\" (UniqueName: \"kubernetes.io/projected/976b7c37-2d5b-4bbe-8214-dad6eb657df0-kube-api-access-wzkbq\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.206755 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.206733    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tmp-dir\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308331 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308292    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-dshm\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308331 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308333    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzkbq\" (UniqueName: \"kubernetes.io/projected/976b7c37-2d5b-4bbe-8214-dad6eb657df0-kube-api-access-wzkbq\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308359    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tmp-dir\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308482    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tls-certs\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308554    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-home\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308793 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308636    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-model-cache\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308869 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308791    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tmp-dir\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.308988 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308961    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-home\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.309113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.308995    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-model-cache\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.310883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.310864    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-dshm\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.311088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.311070    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tls-certs\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.360922 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.360862    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzkbq\" (UniqueName: \"kubernetes.io/projected/976b7c37-2d5b-4bbe-8214-dad6eb657df0-kube-api-access-wzkbq\") pod \"test-model-d4609f-kserve-7fdc7656fc-s2grw\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") " pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.416845 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.416818    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:31.783390 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:31.783361    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"]
Nov 03 13:07:31.784425 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:07:31.784401    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod976b7c37_2d5b_4bbe_8214_dad6eb657df0.slice/crio-9d6ae26a33654520352b046b4fc74912f5342d3c0c19a3b8feb0b33d8bb7f281 WatchSource:0}: Error finding container 9d6ae26a33654520352b046b4fc74912f5342d3c0c19a3b8feb0b33d8bb7f281: Status 404 returned error can't find the container with id 9d6ae26a33654520352b046b4fc74912f5342d3c0c19a3b8feb0b33d8bb7f281
Nov 03 13:07:32.182997 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:32.182903    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw" event={"ID":"976b7c37-2d5b-4bbe-8214-dad6eb657df0","Type":"ContainerStarted","Data":"c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096"}
Nov 03 13:07:32.182997 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:32.182953    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw" event={"ID":"976b7c37-2d5b-4bbe-8214-dad6eb657df0","Type":"ContainerStarted","Data":"9d6ae26a33654520352b046b4fc74912f5342d3c0c19a3b8feb0b33d8bb7f281"}
Nov 03 13:07:32.291819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:32.291772    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw" podStartSLOduration=1.29175857 podStartE2EDuration="1.29175857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:32.288987288 +0000 UTC m=+4601.301453953" watchObservedRunningTime="2026-11-03 13:07:32.29175857 +0000 UTC m=+4601.304225232"
Nov 03 13:07:32.423310 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:32.423260    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:39.564372 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:07:39.564329    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:07:41.417939 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:41.417909    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:41.430753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:41.430726    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:42.256252 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:42.256222    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:07:50.807004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:50.806952    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:07:53.399134 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.399104    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"]
Nov 03 13:07:53.399585 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.399277    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8" podUID="a6868f82-799e-4b22-acac-036fc39682c5" containerName="maas-api" containerID="cri-o://dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962" gracePeriod=30
Nov 03 13:07:53.705149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.705122    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:07:53.730766 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.730738    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-metrics-certs\") pod \"a6868f82-799e-4b22-acac-036fc39682c5\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") "
Nov 03 13:07:53.730929 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.730916    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6868f82-799e-4b22-acac-036fc39682c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6868f82-799e-4b22-acac-036fc39682c5-openshift-service-ca\") pod \"a6868f82-799e-4b22-acac-036fc39682c5\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") "
Nov 03 13:07:53.730994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.730940    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6868f82-799e-4b22-acac-036fc39682c5-kube-api-access-nhkl2\" (UniqueName: \"kubernetes.io/projected/a6868f82-799e-4b22-acac-036fc39682c5-kube-api-access-nhkl2\") pod \"a6868f82-799e-4b22-acac-036fc39682c5\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") "
Nov 03 13:07:53.730994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.730963    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls\") pod \"a6868f82-799e-4b22-acac-036fc39682c5\" (UID: \"a6868f82-799e-4b22-acac-036fc39682c5\") "
Nov 03 13:07:53.731339 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.731304    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6868f82-799e-4b22-acac-036fc39682c5-openshift-service-ca" pod "a6868f82-799e-4b22-acac-036fc39682c5" (UID: "a6868f82-799e-4b22-acac-036fc39682c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:07:53.731518 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.731500    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6868f82-799e-4b22-acac-036fc39682c5-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:53.733182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.733152    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls" pod "a6868f82-799e-4b22-acac-036fc39682c5" (UID: "a6868f82-799e-4b22-acac-036fc39682c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:07:53.733286 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.733194    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-metrics-certs" pod "a6868f82-799e-4b22-acac-036fc39682c5" (UID: "a6868f82-799e-4b22-acac-036fc39682c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:07:53.733483 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.733466    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6868f82-799e-4b22-acac-036fc39682c5-kube-api-access-nhkl2" pod "a6868f82-799e-4b22-acac-036fc39682c5" (UID: "a6868f82-799e-4b22-acac-036fc39682c5"). InnerVolumeSpecName "kube-api-access-nhkl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:07:53.833349 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.833309    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhkl2\" (UniqueName: \"kubernetes.io/projected/a6868f82-799e-4b22-acac-036fc39682c5-kube-api-access-nhkl2\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:53.833349 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.833353    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:53.833576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:53.833370    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6868f82-799e-4b22-acac-036fc39682c5-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:54.372063 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.372026    4909 generic.go:356] "Generic (PLEG): container finished" podID="a6868f82-799e-4b22-acac-036fc39682c5" containerID="dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962" exitCode=0
Nov 03 13:07:54.372341 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.372096    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8" event={"ID":"a6868f82-799e-4b22-acac-036fc39682c5","Type":"ContainerDied","Data":"dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962"}
Nov 03 13:07:54.372532 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.372511    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8" event={"ID":"a6868f82-799e-4b22-acac-036fc39682c5","Type":"ContainerDied","Data":"45f1a9d789dd4a51725f19b4d2bb86097cad12d68a84da2963987235173efb3a"}
Nov 03 13:07:54.372632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.372553    4909 scope.go:122] "RemoveContainer" containerID="dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962"
Nov 03 13:07:54.372711 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.372125    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"
Nov 03 13:07:54.393983 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.393963    4909 scope.go:122] "RemoveContainer" containerID="dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962"
Nov 03 13:07:54.394300 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:07:54.394280    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962\": container with ID starting with dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962 not found: ID does not exist" containerID="dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962"
Nov 03 13:07:54.394372 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.394310    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962"} err="failed to get container status \"dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962\": rpc error: code = NotFound desc = could not find container \"dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962\": container with ID starting with dc33762b955d4951031ba2463bf83b5e5630d150fd54b6eb2603fbd34c44b962 not found: ID does not exist"
Nov 03 13:07:54.527312 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.527283    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"]
Nov 03 13:07:54.573663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:54.573634    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bd346ef-5d6788d94d-58db8"]
Nov 03 13:07:55.124943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:07:55.124909    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6868f82-799e-4b22-acac-036fc39682c5" path="/var/lib/kubelet/pods/a6868f82-799e-4b22-acac-036fc39682c5/volumes"
Nov 03 13:08:09.564172 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:08:09.564094    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:08:16.354781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:16.354743    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"]
Nov 03 13:08:16.400690 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:16.400652    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"]
Nov 03 13:08:28.406608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:28.406572    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"]
Nov 03 13:08:28.506653 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:28.506622    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"]
Nov 03 13:08:31.376057 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.375992    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main" containerID="cri-o://f7b6a7b6f12d83ebeb4b60a1e70a7f2aae796749cc3d606e6fd206e4519e30e0" gracePeriod=45
Nov 03 13:08:31.421683 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.421649    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" containerID="cri-o://2e20e1bdb062325531acc08d346f48878e57184f84a64a71b2792084c719211c" gracePeriod=45
Nov 03 13:08:31.659572 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.659488    4909 generic.go:356] "Generic (PLEG): container finished" podID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerID="2e20e1bdb062325531acc08d346f48878e57184f84a64a71b2792084c719211c" exitCode=0
Nov 03 13:08:31.659765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.659579    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" event={"ID":"fb0a58e5-d12c-4e1e-8015-925a4d85ed35","Type":"ContainerDied","Data":"2e20e1bdb062325531acc08d346f48878e57184f84a64a71b2792084c719211c"}
Nov 03 13:08:31.660895 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.660858    4909 generic.go:356] "Generic (PLEG): container finished" podID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerID="f7b6a7b6f12d83ebeb4b60a1e70a7f2aae796749cc3d606e6fd206e4519e30e0" exitCode=0
Nov 03 13:08:31.661036 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.660957    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" event={"ID":"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2","Type":"ContainerDied","Data":"f7b6a7b6f12d83ebeb4b60a1e70a7f2aae796749cc3d606e6fd206e4519e30e0"}
Nov 03 13:08:31.781986 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.781955    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 13:08:31.784642 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.784618    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 13:08:31.853644 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.853613    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-kube-api-access-47bm6\" (UniqueName: \"kubernetes.io/projected/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-kube-api-access-47bm6\") pod \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") "
Nov 03 13:08:31.853989 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.853955    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-dshm\") pod \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") "
Nov 03 13:08:31.854172 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854144    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-model-cache\") pod \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") "
Nov 03 13:08:31.854347 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854324    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-kube-api-access-b4dqp\" (UniqueName: \"kubernetes.io/projected/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-kube-api-access-b4dqp\") pod \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") "
Nov 03 13:08:31.854536 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854495    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tls-certs\") pod \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") "
Nov 03 13:08:31.854663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854556    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-dshm\") pod \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") "
Nov 03 13:08:31.854663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854603    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-home\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-home\") pod \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") "
Nov 03 13:08:31.854846 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854691    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tls-certs\") pod \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") "
Nov 03 13:08:31.854846 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854736    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tmp-dir\") pod \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") "
Nov 03 13:08:31.854846 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854828    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-home\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-home\") pod \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") "
Nov 03 13:08:31.855026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854943    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tmp-dir\") pod \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\" (UID: \"fb0a58e5-d12c-4e1e-8015-925a4d85ed35\") "
Nov 03 13:08:31.855026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.854978    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-model-cache\") pod \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\" (UID: \"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2\") "
Nov 03 13:08:31.855834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.855375    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-home" pod "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" (UID: "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.855834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.855520    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tmp-dir" pod "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" (UID: "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.855834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.855592    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.855834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.855658    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-home" pod "fb0a58e5-d12c-4e1e-8015-925a4d85ed35" (UID: "fb0a58e5-d12c-4e1e-8015-925a4d85ed35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.855834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.855774    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-model-cache" pod "fb0a58e5-d12c-4e1e-8015-925a4d85ed35" (UID: "fb0a58e5-d12c-4e1e-8015-925a4d85ed35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.855834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.855784    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-model-cache" pod "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" (UID: "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.856252 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.855963    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tmp-dir" pod "fb0a58e5-d12c-4e1e-8015-925a4d85ed35" (UID: "fb0a58e5-d12c-4e1e-8015-925a4d85ed35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.859283 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.859260    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tls-certs" pod "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" (UID: "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:08:31.860092 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.859565    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-kube-api-access-b4dqp" pod "fb0a58e5-d12c-4e1e-8015-925a4d85ed35" (UID: "fb0a58e5-d12c-4e1e-8015-925a4d85ed35"). InnerVolumeSpecName "kube-api-access-b4dqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:31.863642 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.863613    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-dshm" pod "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" (UID: "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.863773 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.863642    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-kube-api-access-47bm6" pod "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" (UID: "e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2"). InnerVolumeSpecName "kube-api-access-47bm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:31.864054 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.864029    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-dshm" pod "fb0a58e5-d12c-4e1e-8015-925a4d85ed35" (UID: "fb0a58e5-d12c-4e1e-8015-925a4d85ed35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.864900 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.864875    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tls-certs" pod "fb0a58e5-d12c-4e1e-8015-925a4d85ed35" (UID: "fb0a58e5-d12c-4e1e-8015-925a4d85ed35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:08:31.957006 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.956975    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957006 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957005    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957019    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957031    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957043    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957054    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47bm6\" (UniqueName: \"kubernetes.io/projected/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-kube-api-access-47bm6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957068    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957081    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957092    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4dqp\" (UniqueName: \"kubernetes.io/projected/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-kube-api-access-b4dqp\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957105    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0a58e5-d12c-4e1e-8015-925a4d85ed35-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.957245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:31.957119    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:32.669355 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.669232    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"
Nov 03 13:08:32.669355 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.669238    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" event={"ID":"fb0a58e5-d12c-4e1e-8015-925a4d85ed35","Type":"ContainerDied","Data":"19a02d2120c639d2fdab89427035d2017c991e6c9d1de3ec94909bdb7303607f"}
Nov 03 13:08:32.669355 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.669290    4909 scope.go:122] "RemoveContainer" containerID="2e20e1bdb062325531acc08d346f48878e57184f84a64a71b2792084c719211c"
Nov 03 13:08:32.670880 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.670863    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"
Nov 03 13:08:32.670964 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.670901    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t" event={"ID":"e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2","Type":"ContainerDied","Data":"3cb64b8a2aa9eb2600cb71f674846f8bf16001a4f9d4e22259b9318714a56471"}
Nov 03 13:08:32.680202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.680176    4909 scope.go:122] "RemoveContainer" containerID="f7b6a7b6f12d83ebeb4b60a1e70a7f2aae796749cc3d606e6fd206e4519e30e0"
Nov 03 13:08:32.683321 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.683261    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:08:32.827004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.826967    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"]
Nov 03 13:08:32.831598 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.831571    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-2-w4-8d2c7c-kserve-59895c555f-g6t9t"]
Nov 03 13:08:32.934795 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.934707    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"]
Nov 03 13:08:32.938975 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:32.938946    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/distinct-w4-8d2c7c-kserve-6cb4c97d7b-s76h9"]
Nov 03 13:08:33.124494 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:33.124460    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" path="/var/lib/kubelet/pods/e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2/volumes"
Nov 03 13:08:33.124929 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:33.124916    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" path="/var/lib/kubelet/pods/fb0a58e5-d12c-4e1e-8015-925a4d85ed35/volumes"
Nov 03 13:08:33.407551 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:33.407492    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" podUID="9b7b7280-6313-425a-b48f-1a2f19666a74" containerName="payload-processing" containerID="cri-o://749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe" gracePeriod=25
Nov 03 13:08:33.507250 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:33.507198    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerName="payload-pre-processing" containerID="cri-o://fabd6a23b3f0c47513485afa3835cd70b507b32c99e03633e8a3e3d9996246a6" gracePeriod=25
Nov 03 13:08:33.706092 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:33.706060    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:08:39.564439 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:08:39.564403    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:08:42.699238 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:42.699203    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" podUID="9b7b7280-6313-425a-b48f-1a2f19666a74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:08:43.706554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:43.706519    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:08:52.699181 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:52.699147    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" podUID="9b7b7280-6313-425a-b48f-1a2f19666a74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:08:53.703076 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.702750    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 13:08:53.805576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.805549    4909 generic.go:356] "Generic (PLEG): container finished" podID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerID="fabd6a23b3f0c47513485afa3835cd70b507b32c99e03633e8a3e3d9996246a6" exitCode=0
Nov 03 13:08:53.805727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.805620    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" event={"ID":"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58","Type":"ContainerDied","Data":"fabd6a23b3f0c47513485afa3835cd70b507b32c99e03633e8a3e3d9996246a6"}
Nov 03 13:08:53.805727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.805673    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" event={"ID":"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58","Type":"ContainerDied","Data":"07dc4ea0c66b1fecb08b38ac879de29e762db6862cb33655a89f6cfc9d61c945"}
Nov 03 13:08:53.805727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.805720    4909 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07dc4ea0c66b1fecb08b38ac879de29e762db6862cb33655a89f6cfc9d61c945"
Nov 03 13:08:53.805960 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.805943    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 13:08:53.807158 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.807136    4909 generic.go:356] "Generic (PLEG): container finished" podID="9b7b7280-6313-425a-b48f-1a2f19666a74" containerID="749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe" exitCode=0
Nov 03 13:08:53.807257 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.807180    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"
Nov 03 13:08:53.807257 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.807198    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" event={"ID":"9b7b7280-6313-425a-b48f-1a2f19666a74","Type":"ContainerDied","Data":"749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe"}
Nov 03 13:08:53.807257 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.807230    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89" event={"ID":"9b7b7280-6313-425a-b48f-1a2f19666a74","Type":"ContainerDied","Data":"95821de31f3aaf041527365357e359b1dcc914e9cae53f128cac220c46308319"}
Nov 03 13:08:53.807257 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.807254    4909 scope.go:122] "RemoveContainer" containerID="749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe"
Nov 03 13:08:53.819168 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.819128    4909 scope.go:122] "RemoveContainer" containerID="749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe"
Nov 03 13:08:53.819495 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:08:53.819473    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe\": container with ID starting with 749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe not found: ID does not exist" containerID="749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe"
Nov 03 13:08:53.819570 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.819502    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe"} err="failed to get container status \"749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe\": rpc error: code = NotFound desc = could not find container \"749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe\": container with ID starting with 749ba9b4787f7d91f7d7a27bc6bb1fcc9089386aaa5b518297f88e7d67cce4fe not found: ID does not exist"
Nov 03 13:08:53.823131 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.823105    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7b7280-6313-425a-b48f-1a2f19666a74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b7280-6313-425a-b48f-1a2f19666a74-plugins-config-volume\") pod \"9b7b7280-6313-425a-b48f-1a2f19666a74\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") "
Nov 03 13:08:53.823249 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.823223    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7b7280-6313-425a-b48f-1a2f19666a74-kube-api-access-9jx9k\" (UniqueName: \"kubernetes.io/projected/9b7b7280-6313-425a-b48f-1a2f19666a74-kube-api-access-9jx9k\") pod \"9b7b7280-6313-425a-b48f-1a2f19666a74\" (UID: \"9b7b7280-6313-425a-b48f-1a2f19666a74\") "
Nov 03 13:08:53.823582 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.823553    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7b7280-6313-425a-b48f-1a2f19666a74-plugins-config-volume" pod "9b7b7280-6313-425a-b48f-1a2f19666a74" (UID: "9b7b7280-6313-425a-b48f-1a2f19666a74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:08:53.823919 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.823896    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b7280-6313-425a-b48f-1a2f19666a74-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:53.884576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.882666    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7b7280-6313-425a-b48f-1a2f19666a74-kube-api-access-9jx9k" pod "9b7b7280-6313-425a-b48f-1a2f19666a74" (UID: "9b7b7280-6313-425a-b48f-1a2f19666a74"). InnerVolumeSpecName "kube-api-access-9jx9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:53.925279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.925197    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-kube-api-access-dx8zp\" (UniqueName: \"kubernetes.io/projected/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-kube-api-access-dx8zp\") pod \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") "
Nov 03 13:08:53.925427 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.925292    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-plugins-config-volume\") pod \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\" (UID: \"fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58\") "
Nov 03 13:08:53.925686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.925655    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-plugins-config-volume" pod "fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" (UID: "fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:08:53.925969 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.925948    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jx9k\" (UniqueName: \"kubernetes.io/projected/9b7b7280-6313-425a-b48f-1a2f19666a74-kube-api-access-9jx9k\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:53.926018 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.925975    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:53.928374 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:53.928349    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-kube-api-access-dx8zp" pod "fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" (UID: "fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58"). InnerVolumeSpecName "kube-api-access-dx8zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:54.027562 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.027516    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8zp\" (UniqueName: \"kubernetes.io/projected/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58-kube-api-access-dx8zp\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:54.218863 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.218829    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"]
Nov 03 13:08:54.222016 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.221989    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d2c7c-59d48648f-mtp89"]
Nov 03 13:08:54.705937 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.705896    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.50:50052\" within 1s: context deadline exceeded"
Nov 03 13:08:54.706351 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:08:54.705986    4909 logging.go:55] [core] [Channel #12104 SubChannel #12105]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.50:50052", ServerName: "10.132.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.50:50052: operation was canceled"
Nov 03 13:08:54.706351 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.705991    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 13:08:54.814300 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.814270    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"
Nov 03 13:08:54.965715 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.965628    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"]
Nov 03 13:08:54.969253 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:54.969226    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d2c7c-66db6597b7-2zk4k"]
Nov 03 13:08:55.123892 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:55.123860    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7b7280-6313-425a-b48f-1a2f19666a74" path="/var/lib/kubelet/pods/9b7b7280-6313-425a-b48f-1a2f19666a74/volumes"
Nov 03 13:08:55.124344 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:55.124329    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" path="/var/lib/kubelet/pods/fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58/volumes"
Nov 03 13:08:59.697208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:08:59.697174    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:09:05.701035 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:05.700995    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"]
Nov 03 13:09:05.701532 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:05.701229    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy" containerID="cri-o://b958e01c9e94edea55c691b336aa73e1ae77b79d4d2cbae9631761313327637e" gracePeriod=30
Nov 03 13:09:09.564590 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:09:09.564553    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:09:10.959083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:10.959055    4909 generic.go:356] "Generic (PLEG): container finished" podID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerID="b958e01c9e94edea55c691b336aa73e1ae77b79d4d2cbae9631761313327637e" exitCode=0
Nov 03 13:09:10.959392 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:10.959092    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" event={"ID":"de3e0234-e011-4964-80e6-5e34e32ec3fe","Type":"ContainerDied","Data":"b958e01c9e94edea55c691b336aa73e1ae77b79d4d2cbae9631761313327637e"}
Nov 03 13:09:10.959392 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:10.959133    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk" event={"ID":"de3e0234-e011-4964-80e6-5e34e32ec3fe","Type":"ContainerDied","Data":"99590baa8dc63cbaae895561cb8d74a989ed82ef76f59687a789cc6af2d4d106"}
Nov 03 13:09:10.959392 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:10.959151    4909 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99590baa8dc63cbaae895561cb8d74a989ed82ef76f59687a789cc6af2d4d106"
Nov 03 13:09:10.965919 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:10.965898    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 13:09:11.053492 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053434    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-credential-socket\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.053492 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053488    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-envoy\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.053663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053506    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-socket\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.053663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053550    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-token\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-token\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.053663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053596    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-kube-api-access-gqk8l\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-kube-api-access-gqk8l\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.053663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053614    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-podinfo\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.054004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053736    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-certs\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.054004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053759    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-credential-socket" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:11.054004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053783    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-data\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.054004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053816    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3e0234-e011-4964-80e6-5e34e32ec3fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3e0234-e011-4964-80e6-5e34e32ec3fe-istiod-ca-cert\") pod \"de3e0234-e011-4964-80e6-5e34e32ec3fe\" (UID: \"de3e0234-e011-4964-80e6-5e34e32ec3fe\") "
Nov 03 13:09:11.054004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053894    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-socket" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:11.054004 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.053980    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-certs" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:11.054342 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.054120    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-data" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:11.054401 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.054382    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e0234-e011-4964-80e6-5e34e32ec3fe-istiod-ca-cert" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:11.054489 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.054470    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.054549 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.054495    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.054549 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.054510    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3e0234-e011-4964-80e6-5e34e32ec3fe-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.054549 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.054524    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.054549 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.054538    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.055912 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.055890    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-podinfo" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:09:11.055990 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.055966    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-kube-api-access-gqk8l" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "kube-api-access-gqk8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:11.056154 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.056137    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-envoy" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:11.056229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.056193    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-token" pod "de3e0234-e011-4964-80e6-5e34e32ec3fe" (UID: "de3e0234-e011-4964-80e6-5e34e32ec3fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:11.155436 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.155411    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.155436 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.155435    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.155569 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.155446    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqk8l\" (UniqueName: \"kubernetes.io/projected/de3e0234-e011-4964-80e6-5e34e32ec3fe-kube-api-access-gqk8l\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.155569 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.155456    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3e0234-e011-4964-80e6-5e34e32ec3fe-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:11.965630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:11.965591    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"
Nov 03 13:09:12.164583 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:12.164545    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"]
Nov 03 13:09:12.210083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:12.210045    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-qx2mk"]
Nov 03 13:09:13.124649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:13.124615    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" path="/var/lib/kubelet/pods/de3e0234-e011-4964-80e6-5e34e32ec3fe/volumes"
Nov 03 13:09:39.565094 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:09:39.565007    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:09:48.438106 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:48.438067    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:09:52.892593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:52.892562    4909 scope.go:122] "RemoveContainer" containerID="b958e01c9e94edea55c691b336aa73e1ae77b79d4d2cbae9631761313327637e"
Nov 03 13:09:52.904832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:09:52.904811    4909 scope.go:122] "RemoveContainer" containerID="fabd6a23b3f0c47513485afa3835cd70b507b32c99e03633e8a3e3d9996246a6"
Nov 03 13:10:09.564690 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:10:09.564648    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:10:35.069358 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:35.069325    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:10:36.012715 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.012661    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"]
Nov 03 13:10:36.013258 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013241    4909 cpu_manager.go:414] "containerMap: removing container" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerName="payload-pre-processing"
Nov 03 13:10:36.013305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013261    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerName="payload-pre-processing"
Nov 03 13:10:36.013305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013275    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main"
Nov 03 13:10:36.013305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013280    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main"
Nov 03 13:10:36.013305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013303    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a6868f82-799e-4b22-acac-036fc39682c5" containerName="maas-api"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013308    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6868f82-799e-4b22-acac-036fc39682c5" containerName="maas-api"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013324    4909 cpu_manager.go:414] "containerMap: removing container" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013329    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013338    4909 cpu_manager.go:414] "containerMap: removing container" podUID="9b7b7280-6313-425a-b48f-1a2f19666a74" containerName="payload-processing"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013342    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7b7280-6313-425a-b48f-1a2f19666a74" containerName="payload-processing"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013350    4909 cpu_manager.go:414] "containerMap: removing container" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013355    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy"
Nov 03 13:10:36.013438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013439    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4c33dd-116d-4fbc-bfe7-d430bd3d7c58" containerName="payload-pre-processing"
Nov 03 13:10:36.013712 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013449    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cb2d07-e297-42a6-bf7e-5b28b1d4d2f2" containerName="main"
Nov 03 13:10:36.013712 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013459    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6868f82-799e-4b22-acac-036fc39682c5" containerName="maas-api"
Nov 03 13:10:36.013712 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013467    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0a58e5-d12c-4e1e-8015-925a4d85ed35" containerName="main"
Nov 03 13:10:36.013712 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013473    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7b7280-6313-425a-b48f-1a2f19666a74" containerName="payload-processing"
Nov 03 13:10:36.013712 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.013478    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3e0234-e011-4964-80e6-5e34e32ec3fe" containerName="istio-proxy"
Nov 03 13:10:36.016843 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.016820    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.113173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.113139    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"openshift-service-ca.crt\""
Nov 03 13:10:36.113173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.113147    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"default-dockercfg-zl57l\""
Nov 03 13:10:36.113658 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.113192    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"kube-root-ca.crt\""
Nov 03 13:10:36.113658 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.113279    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"sub-test-model-a4489c-kserve-self-signed-certs\""
Nov 03 13:10:36.118274 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.118248    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"]
Nov 03 13:10:36.165056 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.165025    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-kube-api-access-824ts\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.165192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.165105    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tls-certs\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.165192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.165142    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-home\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.165192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.165174    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-model-cache\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.165384 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.165277    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tmp-dir\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.165384 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.165329    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-dshm\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.274804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.274456    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tmp-dir\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.274804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.274542    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-dshm\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.274804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.274649    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-kube-api-access-824ts\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.274804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.274734    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tls-certs\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.274804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.274770    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-home\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.274804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.274803    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-model-cache\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.275395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.275369    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-model-cache\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.275662 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.275639    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tmp-dir\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.289721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.276908    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-home\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.289721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.279282    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-dshm\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.294804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.293805    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tls-certs\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.333392 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.333362    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-kube-api-access-824ts\") pod \"sub-test-model-a4489c-kserve-7865fbb6cf-29n5m\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:36.629386 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:36.629307    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:37.015935 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:37.015903    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"]
Nov 03 13:10:37.017690 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:10:37.017664    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b1569f9_71ca_42c0_b16d_4d59b9cd7764.slice/crio-ec18cb4dda599cd611b326fe614a5363e6afb26c4dd0fbb3db720745dd38be48 WatchSource:0}: Error finding container ec18cb4dda599cd611b326fe614a5363e6afb26c4dd0fbb3db720745dd38be48: Status 404 returned error can't find the container with id ec18cb4dda599cd611b326fe614a5363e6afb26c4dd0fbb3db720745dd38be48
Nov 03 13:10:37.546798 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:37.546760    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m" event={"ID":"4b1569f9-71ca-42c0-b16d-4d59b9cd7764","Type":"ContainerStarted","Data":"0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5"}
Nov 03 13:10:37.546798 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:37.546801    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m" event={"ID":"4b1569f9-71ca-42c0-b16d-4d59b9cd7764","Type":"ContainerStarted","Data":"ec18cb4dda599cd611b326fe614a5363e6afb26c4dd0fbb3db720745dd38be48"}
Nov 03 13:10:37.629629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:37.629603    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:37.659112 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:37.659046    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m" podStartSLOduration=2.659027757 podStartE2EDuration="2.659027757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:37.653882645 +0000 UTC m=+4786.666349308" watchObservedRunningTime="2026-11-03 13:10:37.659027757 +0000 UTC m=+4786.671494424"
Nov 03 13:10:39.565515 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:10:39.565481    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:10:46.630320 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:46.630284    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:46.643124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:46.643097    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:47.620151 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:47.620119    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:10:51.766968 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:51.766942    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:10:51.776370 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:10:51.776349    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:11:04.987976 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:04.987896    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"]
Nov 03 13:11:04.992070 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:04.992048    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.038030 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.037992    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlsj\" (UniqueName: \"kubernetes.io/projected/71fcf678-1da0-47bf-b12d-714d130fc6ea-kube-api-access-lxlsj\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.038222 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.038086    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-home\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.038222 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.038160    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-dshm\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.038222 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.038208    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-tmp-dir\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.038417 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.038242    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-model-cache\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.038417 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.038291    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fcf678-1da0-47bf-b12d-714d130fc6ea-tls-certs\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.040538 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.040511    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"default-dockercfg-zlm8p\""
Nov 03 13:11:05.086598 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.086554    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"openshift-service-ca.crt\""
Nov 03 13:11:05.086598 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.086590    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"kube-root-ca.crt\""
Nov 03 13:11:05.086858 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.086688    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-6fb3a8\"/\"sub-test-model-6fb3a8-kserve-self-signed-certs\""
Nov 03 13:11:05.089625 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.089607    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"]
Nov 03 13:11:05.139178 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139148    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlsj\" (UniqueName: \"kubernetes.io/projected/71fcf678-1da0-47bf-b12d-714d130fc6ea-kube-api-access-lxlsj\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139315 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139206    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-home\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139315 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139258    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-dshm\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139315 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139289    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-tmp-dir\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139315 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139312    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-model-cache\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139462    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fcf678-1da0-47bf-b12d-714d130fc6ea-tls-certs\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139570    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-home\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139709 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139641    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-model-cache\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.139784 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.139768    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-tmp-dir\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.142132 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.142108    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fcf678-1da0-47bf-b12d-714d130fc6ea-tls-certs\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.164799 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.159099    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-dshm\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.195661 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.195589    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlsj\" (UniqueName: \"kubernetes.io/projected/71fcf678-1da0-47bf-b12d-714d130fc6ea-kube-api-access-lxlsj\") pod \"sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") " pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.304659 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.304576    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:05.671127 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.671099    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"]
Nov 03 13:11:05.672827 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:11:05.672796    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71fcf678_1da0_47bf_b12d_714d130fc6ea.slice/crio-0fff81bfbe7eb96b6e63f586419e730b698e67af60d37581bf0b754f18e85b25 WatchSource:0}: Error finding container 0fff81bfbe7eb96b6e63f586419e730b698e67af60d37581bf0b754f18e85b25: Status 404 returned error can't find the container with id 0fff81bfbe7eb96b6e63f586419e730b698e67af60d37581bf0b754f18e85b25
Nov 03 13:11:05.720088 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:05.720055    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9" event={"ID":"71fcf678-1da0-47bf-b12d-714d130fc6ea","Type":"ContainerStarted","Data":"0fff81bfbe7eb96b6e63f586419e730b698e67af60d37581bf0b754f18e85b25"}
Nov 03 13:11:06.742457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:06.742409    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9" event={"ID":"71fcf678-1da0-47bf-b12d-714d130fc6ea","Type":"ContainerStarted","Data":"c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609"}
Nov 03 13:11:06.850048 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:06.849994    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9" podStartSLOduration=2.849979746 podStartE2EDuration="2.849979746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:06.848931098 +0000 UTC m=+4815.861397761" watchObservedRunningTime="2026-11-03 13:11:06.849979746 +0000 UTC m=+4815.862446409"
Nov 03 13:11:07.304862 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:07.304823    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:09.564636 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:11:09.564599    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:11:15.305284 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:15.305246    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:15.317898 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:15.317868    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:15.817391 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:15.817358    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:11:27.218486 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:11:27.218450    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:11:39.564772 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:11:39.564730    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:12:08.444841 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:08.444808    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"]
Nov 03 13:12:09.565364 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:12:09.565326    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:12:21.991910 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:21.991873    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:12:23.465459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.465395    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw" podUID="976b7c37-2d5b-4bbe-8214-dad6eb657df0" containerName="main" containerID="cri-o://c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096" gracePeriod=45
Nov 03 13:12:23.747423 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.747398    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:12:23.866024 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.865988    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tmp-dir\") pod \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") "
Nov 03 13:12:23.866192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.866045    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-home\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-home\") pod \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") "
Nov 03 13:12:23.866192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.866111    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976b7c37-2d5b-4bbe-8214-dad6eb657df0-kube-api-access-wzkbq\" (UniqueName: \"kubernetes.io/projected/976b7c37-2d5b-4bbe-8214-dad6eb657df0-kube-api-access-wzkbq\") pod \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") "
Nov 03 13:12:23.866192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.866180    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tls-certs\" (UniqueName: \"kubernetes.io/secret/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tls-certs\") pod \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") "
Nov 03 13:12:23.866396 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.866228    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-model-cache\") pod \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") "
Nov 03 13:12:23.866396 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.866264    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tmp-dir" pod "976b7c37-2d5b-4bbe-8214-dad6eb657df0" (UID: "976b7c37-2d5b-4bbe-8214-dad6eb657df0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:23.866396 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.866277    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-dshm\") pod \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\" (UID: \"976b7c37-2d5b-4bbe-8214-dad6eb657df0\") "
Nov 03 13:12:23.866961 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.866933    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:23.867255 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.867198    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-model-cache" pod "976b7c37-2d5b-4bbe-8214-dad6eb657df0" (UID: "976b7c37-2d5b-4bbe-8214-dad6eb657df0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:23.867415 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.867380    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-home" pod "976b7c37-2d5b-4bbe-8214-dad6eb657df0" (UID: "976b7c37-2d5b-4bbe-8214-dad6eb657df0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:23.869042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.869020    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-dshm" pod "976b7c37-2d5b-4bbe-8214-dad6eb657df0" (UID: "976b7c37-2d5b-4bbe-8214-dad6eb657df0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:23.869133 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.869057    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976b7c37-2d5b-4bbe-8214-dad6eb657df0-kube-api-access-wzkbq" pod "976b7c37-2d5b-4bbe-8214-dad6eb657df0" (UID: "976b7c37-2d5b-4bbe-8214-dad6eb657df0"). InnerVolumeSpecName "kube-api-access-wzkbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:12:23.869761 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.869729    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tls-certs" pod "976b7c37-2d5b-4bbe-8214-dad6eb657df0" (UID: "976b7c37-2d5b-4bbe-8214-dad6eb657df0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:12:23.967681 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.967647    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:23.967681 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.967679    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzkbq\" (UniqueName: \"kubernetes.io/projected/976b7c37-2d5b-4bbe-8214-dad6eb657df0-kube-api-access-wzkbq\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:23.967681 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.967689    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976b7c37-2d5b-4bbe-8214-dad6eb657df0-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:23.967681 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.967715    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:23.968001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:23.967737    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976b7c37-2d5b-4bbe-8214-dad6eb657df0-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:24.263726 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.263663    4909 generic.go:356] "Generic (PLEG): container finished" podID="976b7c37-2d5b-4bbe-8214-dad6eb657df0" containerID="c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096" exitCode=0
Nov 03 13:12:24.263726 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.263724    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"
Nov 03 13:12:24.263964 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.263726    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw" event={"ID":"976b7c37-2d5b-4bbe-8214-dad6eb657df0","Type":"ContainerDied","Data":"c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096"}
Nov 03 13:12:24.263964 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.263835    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw" event={"ID":"976b7c37-2d5b-4bbe-8214-dad6eb657df0","Type":"ContainerDied","Data":"9d6ae26a33654520352b046b4fc74912f5342d3c0c19a3b8feb0b33d8bb7f281"}
Nov 03 13:12:24.263964 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.263857    4909 scope.go:122] "RemoveContainer" containerID="c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096"
Nov 03 13:12:24.274679 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.274659    4909 scope.go:122] "RemoveContainer" containerID="c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096"
Nov 03 13:12:24.274966 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:12:24.274944    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096\": container with ID starting with c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096 not found: ID does not exist" containerID="c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096"
Nov 03 13:12:24.275019 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.274973    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096"} err="failed to get container status \"c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096\": rpc error: code = NotFound desc = could not find container \"c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096\": container with ID starting with c3db21475910c2b4758811950ac22529a7a34cda2661a66296492662ff593096 not found: ID does not exist"
Nov 03 13:12:24.416101 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.416077    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"]
Nov 03 13:12:24.461955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:24.461930    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d4609f/test-model-d4609f-kserve-7fdc7656fc-s2grw"]
Nov 03 13:12:25.124348 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:25.124315    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976b7c37-2d5b-4bbe-8214-dad6eb657df0" path="/var/lib/kubelet/pods/976b7c37-2d5b-4bbe-8214-dad6eb657df0/volumes"
Nov 03 13:12:39.564261 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:12:39.564171    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:12:50.348963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:50.348921    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"]
Nov 03 13:12:50.446279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:50.446247    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"]
Nov 03 13:12:55.350233 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.350191    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx" podUID="e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" containerName="payload-processing" containerID="cri-o://b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8" gracePeriod=25
Nov 03 13:12:55.446970 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.446914    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" podUID="10e11887-1355-4bad-b890-3a32ab9edf74" containerName="payload-pre-processing" containerID="cri-o://6bebfccfe04710fdd2b6ff3062917cb6a97127a5b9c265046a00d6dc726a1fe5" gracePeriod=25
Nov 03 13:12:55.621123 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.621057    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:12:55.753971 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.753937    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-kube-api-access-7psnd\" (UniqueName: \"kubernetes.io/projected/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-kube-api-access-7psnd\") pod \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") "
Nov 03 13:12:55.754243 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.754217    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-plugins-config-volume\") pod \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\" (UID: \"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2\") "
Nov 03 13:12:55.754543 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.754520    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-plugins-config-volume" pod "e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" (UID: "e2f1b5ed-c0aa-493e-8d31-821d4fc115a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:12:55.754734 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.754712    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:55.756185 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.756163    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-kube-api-access-7psnd" pod "e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" (UID: "e2f1b5ed-c0aa-493e-8d31-821d4fc115a2"). InnerVolumeSpecName "kube-api-access-7psnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:12:55.855648 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:55.855626    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7psnd\" (UniqueName: \"kubernetes.io/projected/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2-kube-api-access-7psnd\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:56.457484 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.457451    4909 generic.go:356] "Generic (PLEG): container finished" podID="e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" containerID="b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8" exitCode=0
Nov 03 13:12:56.457943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.457499    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"
Nov 03 13:12:56.457943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.457530    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx" event={"ID":"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2","Type":"ContainerDied","Data":"b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8"}
Nov 03 13:12:56.457943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.457583    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx" event={"ID":"e2f1b5ed-c0aa-493e-8d31-821d4fc115a2","Type":"ContainerDied","Data":"527ddfc3dbee1b8115bc6f44d877d2f072c5ebe3176e428cab6aa901a8772f46"}
Nov 03 13:12:56.457943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.457605    4909 scope.go:122] "RemoveContainer" containerID="b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8"
Nov 03 13:12:56.467885 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.467869    4909 scope.go:122] "RemoveContainer" containerID="b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8"
Nov 03 13:12:56.468137 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:12:56.468118    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8\": container with ID starting with b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8 not found: ID does not exist" containerID="b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8"
Nov 03 13:12:56.468212 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.468143    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8"} err="failed to get container status \"b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8\": rpc error: code = NotFound desc = could not find container \"b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8\": container with ID starting with b4960be583579235b64c9d1aa3f751d37c9a986569f1f3ac8f260f07ea8b4dc8 not found: ID does not exist"
Nov 03 13:12:56.611949 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.611920    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"]
Nov 03 13:12:56.657988 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:56.657947    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4609f-58479c95f-ft2mx"]
Nov 03 13:12:57.124408 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:12:57.124379    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" path="/var/lib/kubelet/pods/e2f1b5ed-c0aa-493e-8d31-821d4fc115a2/volumes"
Nov 03 13:13:01.487910 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:01.487872    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" podUID="10e11887-1355-4bad-b890-3a32ab9edf74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:13:09.565320 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:13:09.565251    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:13:11.488491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:11.488448    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" podUID="10e11887-1355-4bad-b890-3a32ab9edf74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:13:15.573834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.573808    4909 generic.go:356] "Generic (PLEG): container finished" podID="10e11887-1355-4bad-b890-3a32ab9edf74" containerID="6bebfccfe04710fdd2b6ff3062917cb6a97127a5b9c265046a00d6dc726a1fe5" exitCode=0
Nov 03 13:13:15.574200 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.573882    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" event={"ID":"10e11887-1355-4bad-b890-3a32ab9edf74","Type":"ContainerDied","Data":"6bebfccfe04710fdd2b6ff3062917cb6a97127a5b9c265046a00d6dc726a1fe5"}
Nov 03 13:13:15.726565 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.726536    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:13:15.796592 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.796566    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e11887-1355-4bad-b890-3a32ab9edf74-kube-api-access-dvwnf\" (UniqueName: \"kubernetes.io/projected/10e11887-1355-4bad-b890-3a32ab9edf74-kube-api-access-dvwnf\") pod \"10e11887-1355-4bad-b890-3a32ab9edf74\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") "
Nov 03 13:13:15.796784 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.796739    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e11887-1355-4bad-b890-3a32ab9edf74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e11887-1355-4bad-b890-3a32ab9edf74-plugins-config-volume\") pod \"10e11887-1355-4bad-b890-3a32ab9edf74\" (UID: \"10e11887-1355-4bad-b890-3a32ab9edf74\") "
Nov 03 13:13:15.797111 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.797077    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e11887-1355-4bad-b890-3a32ab9edf74-plugins-config-volume" pod "10e11887-1355-4bad-b890-3a32ab9edf74" (UID: "10e11887-1355-4bad-b890-3a32ab9edf74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:13:15.798673 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.798652    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e11887-1355-4bad-b890-3a32ab9edf74-kube-api-access-dvwnf" pod "10e11887-1355-4bad-b890-3a32ab9edf74" (UID: "10e11887-1355-4bad-b890-3a32ab9edf74"). InnerVolumeSpecName "kube-api-access-dvwnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:15.898597 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.898566    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e11887-1355-4bad-b890-3a32ab9edf74-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:15.898597 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:15.898590    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwnf\" (UniqueName: \"kubernetes.io/projected/10e11887-1355-4bad-b890-3a32ab9edf74-kube-api-access-dvwnf\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:16.293090 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:16.293056    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:13:16.580780 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:16.580686    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"
Nov 03 13:13:16.581216 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:16.580688    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx" event={"ID":"10e11887-1355-4bad-b890-3a32ab9edf74","Type":"ContainerDied","Data":"b5eb8521e856528cf2b6bfe0e4946a4d36372940b6a0b69e26da8560a10d8aed"}
Nov 03 13:13:16.581216 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:16.580853    4909 scope.go:122] "RemoveContainer" containerID="6bebfccfe04710fdd2b6ff3062917cb6a97127a5b9c265046a00d6dc726a1fe5"
Nov 03 13:13:16.733797 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:16.733759    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"]
Nov 03 13:13:16.738535 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:16.738506    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4609f-5685d49867-bgmcx"]
Nov 03 13:13:17.123275 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:17.123243    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e11887-1355-4bad-b890-3a32ab9edf74" path="/var/lib/kubelet/pods/10e11887-1355-4bad-b890-3a32ab9edf74/volumes"
Nov 03 13:13:29.831718 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:29.831665    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"]
Nov 03 13:13:29.834227 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:29.831922    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz" podUID="96ca5fbc-b2cc-4e72-b046-9c94854cf60a" containerName="maas-api" containerID="cri-o://046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43" gracePeriod=30
Nov 03 13:13:30.142236 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.142206    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:13:30.269246 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.269218    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-metrics-certs\") pod \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") "
Nov 03 13:13:30.269421 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.269299    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-openshift-service-ca\") pod \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") "
Nov 03 13:13:30.269421 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.269359    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls\") pod \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") "
Nov 03 13:13:30.269622 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.269601    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-kube-api-access-n7nzp\" (UniqueName: \"kubernetes.io/projected/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-kube-api-access-n7nzp\") pod \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\" (UID: \"96ca5fbc-b2cc-4e72-b046-9c94854cf60a\") "
Nov 03 13:13:30.269769 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.269746    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-openshift-service-ca" pod "96ca5fbc-b2cc-4e72-b046-9c94854cf60a" (UID: "96ca5fbc-b2cc-4e72-b046-9c94854cf60a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:13:30.270277 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.270257    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.272047 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.272021    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-metrics-certs" pod "96ca5fbc-b2cc-4e72-b046-9c94854cf60a" (UID: "96ca5fbc-b2cc-4e72-b046-9c94854cf60a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:13:30.272149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.272043    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls" pod "96ca5fbc-b2cc-4e72-b046-9c94854cf60a" (UID: "96ca5fbc-b2cc-4e72-b046-9c94854cf60a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:13:30.272335 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.272315    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-kube-api-access-n7nzp" pod "96ca5fbc-b2cc-4e72-b046-9c94854cf60a" (UID: "96ca5fbc-b2cc-4e72-b046-9c94854cf60a"). InnerVolumeSpecName "kube-api-access-n7nzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:30.384678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.384541    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7nzp\" (UniqueName: \"kubernetes.io/projected/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-kube-api-access-n7nzp\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.384678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.384583    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.384678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.384599    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96ca5fbc-b2cc-4e72-b046-9c94854cf60a-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.668826 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.668744    4909 generic.go:356] "Generic (PLEG): container finished" podID="96ca5fbc-b2cc-4e72-b046-9c94854cf60a" containerID="046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43" exitCode=0
Nov 03 13:13:30.668826 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.668787    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"
Nov 03 13:13:30.669026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.668835    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz" event={"ID":"96ca5fbc-b2cc-4e72-b046-9c94854cf60a","Type":"ContainerDied","Data":"046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43"}
Nov 03 13:13:30.669026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.668893    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz" event={"ID":"96ca5fbc-b2cc-4e72-b046-9c94854cf60a","Type":"ContainerDied","Data":"a7456b8105098e2aa40ad1d49f3c927d63ab1da6c3094d98f9a3caf6f5f61da4"}
Nov 03 13:13:30.669026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.668915    4909 scope.go:122] "RemoveContainer" containerID="046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43"
Nov 03 13:13:30.721296 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.683598    4909 scope.go:122] "RemoveContainer" containerID="046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43"
Nov 03 13:13:30.721296 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:13:30.684083    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43\": container with ID starting with 046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43 not found: ID does not exist" containerID="046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43"
Nov 03 13:13:30.721296 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.684122    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43"} err="failed to get container status \"046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43\": rpc error: code = NotFound desc = could not find container \"046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43\": container with ID starting with 046fbad97b1a4cfae5d78cc63ae4b7649f62335670d5074a397cfd843d7e5e43 not found: ID does not exist"
Nov 03 13:13:30.839221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.839190    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"]
Nov 03 13:13:30.885044 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:30.885009    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d4609f-7b9984fc8c-hpblz"]
Nov 03 13:13:31.124175 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:31.124143    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ca5fbc-b2cc-4e72-b046-9c94854cf60a" path="/var/lib/kubelet/pods/96ca5fbc-b2cc-4e72-b046-9c94854cf60a/volumes"
Nov 03 13:13:37.309978 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:37.309941    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"]
Nov 03 13:13:37.310508 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:37.310178    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy" containerID="cri-o://6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05" gracePeriod=30
Nov 03 13:13:39.564773 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:13:39.564678    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:13:42.584843 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.584784    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:13:42.696489 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696458    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-envoy\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.696676 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696539    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-socket\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.696676 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696611    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-token\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-token\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.696676 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696651    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-credential-socket\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.696883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696744    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-podinfo\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.696883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696781    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-kube-api-access-hsncw\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-kube-api-access-hsncw\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.696883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696813    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-certs\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.696883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696817    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-socket" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:42.697104 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696897    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-data\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.697104 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696948    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283a09fd-6d12-43f2-8672-815f5f4d7863-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a09fd-6d12-43f2-8672-815f5f4d7863-istiod-ca-cert\") pod \"283a09fd-6d12-43f2-8672-815f5f4d7863\" (UID: \"283a09fd-6d12-43f2-8672-815f5f4d7863\") "
Nov 03 13:13:42.697104 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.696977    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-credential-socket" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:42.697104 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.697072    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-certs" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:42.697326 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.697230    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-data" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:42.697487 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.697464    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.697579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.697491    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.697579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.697507    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.697579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.697520    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.697579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.697521    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283a09fd-6d12-43f2-8672-815f5f4d7863-istiod-ca-cert" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:13:42.699007 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.698983    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-envoy" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:42.699132 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.699050    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-token" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:42.699416 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.699397    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-kube-api-access-hsncw" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "kube-api-access-hsncw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:42.699488 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.699433    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-podinfo" pod "283a09fd-6d12-43f2-8672-815f5f4d7863" (UID: "283a09fd-6d12-43f2-8672-815f5f4d7863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:13:42.746604 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.746573    4909 generic.go:356] "Generic (PLEG): container finished" podID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerID="6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05" exitCode=0
Nov 03 13:13:42.746757 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.746631    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"
Nov 03 13:13:42.746757 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.746652    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" event={"ID":"283a09fd-6d12-43f2-8672-815f5f4d7863","Type":"ContainerDied","Data":"6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05"}
Nov 03 13:13:42.746757 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.746749    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz" event={"ID":"283a09fd-6d12-43f2-8672-815f5f4d7863","Type":"ContainerDied","Data":"b2713c56b3d57402b8ae8fecad24b983b4e71ad7d421fccbe8aa185a5e21d631"}
Nov 03 13:13:42.746935 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.746776    4909 scope.go:122] "RemoveContainer" containerID="6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05"
Nov 03 13:13:42.760068 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.760049    4909 scope.go:122] "RemoveContainer" containerID="8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43"
Nov 03 13:13:42.768850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.768827    4909 scope.go:122] "RemoveContainer" containerID="6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05"
Nov 03 13:13:42.769150 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:13:42.769131    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05\": container with ID starting with 6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05 not found: ID does not exist" containerID="6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05"
Nov 03 13:13:42.769218 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.769159    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05"} err="failed to get container status \"6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05\": rpc error: code = NotFound desc = could not find container \"6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05\": container with ID starting with 6cb3dedcf3ba456d74a956b62ed4a88df9313e00384d6013c89499f0d1d0bc05 not found: ID does not exist"
Nov 03 13:13:42.769218 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.769176    4909 scope.go:122] "RemoveContainer" containerID="8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43"
Nov 03 13:13:42.769425 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:13:42.769407    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43\": container with ID starting with 8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43 not found: ID does not exist" containerID="8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43"
Nov 03 13:13:42.769465 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.769434    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43"} err="failed to get container status \"8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43\": rpc error: code = NotFound desc = could not find container \"8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43\": container with ID starting with 8f36fa44746e0d5ede59dda947a1e937933c11ed329c8c98e00220734268ee43 not found: ID does not exist"
Nov 03 13:13:42.798862 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.798836    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.798862 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.798858    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.799001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.798867    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsncw\" (UniqueName: \"kubernetes.io/projected/283a09fd-6d12-43f2-8672-815f5f4d7863-kube-api-access-hsncw\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.799001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.798877    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a09fd-6d12-43f2-8672-815f5f4d7863-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.799001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.798887    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a09fd-6d12-43f2-8672-815f5f4d7863-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:42.900266 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.900196    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"]
Nov 03 13:13:42.903017 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:42.902992    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d4609f-openshift-default-6bc776fc7-gb6zz"]
Nov 03 13:13:43.123958 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:43.123923    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" path="/var/lib/kubelet/pods/283a09fd-6d12-43f2-8672-815f5f4d7863/volumes"
Nov 03 13:13:49.901144 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901107    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"]
Nov 03 13:13:49.901737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901718    4909 cpu_manager.go:414] "containerMap: removing container" podUID="976b7c37-2d5b-4bbe-8214-dad6eb657df0" containerName="main"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901743    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976b7c37-2d5b-4bbe-8214-dad6eb657df0" containerName="main"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901768    4909 cpu_manager.go:414] "containerMap: removing container" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901773    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901793    4909 cpu_manager.go:414] "containerMap: removing container" podUID="10e11887-1355-4bad-b890-3a32ab9edf74" containerName="payload-pre-processing"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901801    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e11887-1355-4bad-b890-3a32ab9edf74" containerName="payload-pre-processing"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901811    4909 cpu_manager.go:414] "containerMap: removing container" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901816    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901829    4909 cpu_manager.go:414] "containerMap: removing container" podUID="96ca5fbc-b2cc-4e72-b046-9c94854cf60a" containerName="maas-api"
Nov 03 13:13:49.901844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901833    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ca5fbc-b2cc-4e72-b046-9c94854cf60a" containerName="maas-api"
Nov 03 13:13:49.902433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901853    4909 cpu_manager.go:414] "containerMap: removing container" podUID="e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" containerName="payload-processing"
Nov 03 13:13:49.902433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901858    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" containerName="payload-processing"
Nov 03 13:13:49.902433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901962    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f1b5ed-c0aa-493e-8d31-821d4fc115a2" containerName="payload-processing"
Nov 03 13:13:49.902433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901971    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="976b7c37-2d5b-4bbe-8214-dad6eb657df0" containerName="main"
Nov 03 13:13:49.902433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901981    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy"
Nov 03 13:13:49.902433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901988    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e11887-1355-4bad-b890-3a32ab9edf74" containerName="payload-pre-processing"
Nov 03 13:13:49.902433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.901996    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ca5fbc-b2cc-4e72-b046-9c94854cf60a" containerName="maas-api"
Nov 03 13:13:49.905391 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.905370    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.991582 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.991539    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-data\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.991899 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.991868    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-credential-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.992055 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.992037    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-certs\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.992189 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.992159    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2059f309-d7d6-4d00-b3cb-4d554b564068-istiod-ca-cert\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.992319 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.992212    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-token\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.992319 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.992296    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-envoy\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.992431 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.992374    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.992483 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.992439    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-podinfo\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.992539 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.992486    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhk6q\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-kube-api-access-dhk6q\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:49.994181 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:49.994155    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"]
Nov 03 13:13:50.044019 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.043984    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1250b3-openshift-default-dockercfg-sksxt\""
Nov 03 13:13:50.093421 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093392    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-podinfo\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.093579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093444    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhk6q\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-kube-api-access-dhk6q\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.093579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093492    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-data\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.093579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093515    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-credential-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.093579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093555    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-certs\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.093834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093595    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2059f309-d7d6-4d00-b3cb-4d554b564068-istiod-ca-cert\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.093834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093634    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-token\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.093834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093756    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-envoy\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.094010 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093846    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.094010 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093971    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-credential-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.094010 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.093997    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-certs\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.094281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.094259    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.094459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.094441    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-data\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.094459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.094449    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2059f309-d7d6-4d00-b3cb-4d554b564068-istiod-ca-cert\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.096233 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.096208    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-podinfo\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.096545 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.096522    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-envoy\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.146927 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.146901    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhk6q\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-kube-api-access-dhk6q\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.147113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.147091    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-token\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.219253 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.219219    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:50.632796 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.632766    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"]
Nov 03 13:13:50.634159 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:13:50.634131    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2059f309_d7d6_4d00_b3cb_4d554b564068.slice/crio-3d4b28d2a8528d87faef185042206b3e892f9d3aafd587bc6e85e99a3100763c WatchSource:0}: Error finding container 3d4b28d2a8528d87faef185042206b3e892f9d3aafd587bc6e85e99a3100763c: Status 404 returned error can't find the container with id 3d4b28d2a8528d87faef185042206b3e892f9d3aafd587bc6e85e99a3100763c
Nov 03 13:13:50.636751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.636716    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:13:50.636827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.636785    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:13:50.636827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.636816    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:13:50.801522 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.801463    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" event={"ID":"2059f309-d7d6-4d00-b3cb-4d554b564068","Type":"ContainerStarted","Data":"3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0"}
Nov 03 13:13:50.801731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.801552    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" event={"ID":"2059f309-d7d6-4d00-b3cb-4d554b564068","Type":"ContainerStarted","Data":"3d4b28d2a8528d87faef185042206b3e892f9d3aafd587bc6e85e99a3100763c"}
Nov 03 13:13:50.909991 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:50.909897    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podStartSLOduration=1.909880995 podStartE2EDuration="1.909880995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:50.908580875 +0000 UTC m=+4979.921047548" watchObservedRunningTime="2026-11-03 13:13:50.909880995 +0000 UTC m=+4979.922347661"
Nov 03 13:13:51.219902 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:51.219874    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:51.221536 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:51.221509    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 13:13:51.221664 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:51.221558    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 13:13:52.219542 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:52.219506    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 13:13:52.219972 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:52.219560    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 13:13:53.220072 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:53.220039    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 13:13:53.220562 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:53.220095    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 13:13:54.219857 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:54.219820    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 13:13:54.220025 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:54.219874    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 13:13:54.884636 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:54.884595    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:13:55.220250 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:55.220217    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 13:13:55.220430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:55.220268    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 13:13:56.220413 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:56.220377    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 13:13:56.220837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:56.220432    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 13:13:57.301338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:57.301009    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:57.301883 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:57.301531    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:13:57.302395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:13:57.302366    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:14:09.566182 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:14:09.566099    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:14:21.929031 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:21.928996    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"]
Nov 03 13:14:29.063444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:29.063407    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"]
Nov 03 13:14:29.063870 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:29.063616    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy" containerID="cri-o://d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd" gracePeriod=30
Nov 03 13:14:33.062066 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:33.062032    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:14:34.368743 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.368707    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 13:14:34.472955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.472924    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-credential-socket\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.472961    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-socket\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473043    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-token\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-token\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473081    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-envoy\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473138    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-data\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473188    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istiod-ca-cert\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473216    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-credential-socket" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:34.473364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473266    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-certs\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473287    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-socket" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:34.473364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473315    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-kube-api-access-cf9p2\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-kube-api-access-cf9p2\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473414    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-podinfo\") pod \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\" (UID: \"2bf516db-3ff5-4e54-b0f4-2bf041f6b161\") "
Nov 03 13:14:34.473744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473446    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-data" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:34.473744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473533    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-certs" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:34.473744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473634    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istiod-ca-cert" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:34.474002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473915    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.474002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473938    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.474002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473953    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.474002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473966    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.474002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.473979    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.475402 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.475380    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-token" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:34.475402 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.475384    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-envoy" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:34.475543 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.475515    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-kube-api-access-cf9p2" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "kube-api-access-cf9p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:34.475543 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.475529    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-podinfo" pod "2bf516db-3ff5-4e54-b0f4-2bf041f6b161" (UID: "2bf516db-3ff5-4e54-b0f4-2bf041f6b161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:14:34.575122 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.575089    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9p2\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-kube-api-access-cf9p2\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.575122 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.575118    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.575122 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.575128    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:34.575331 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:34.575138    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf516db-3ff5-4e54-b0f4-2bf041f6b161-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:35.133443 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.133415    4909 generic.go:356] "Generic (PLEG): container finished" podID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerID="d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd" exitCode=0
Nov 03 13:14:35.133574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.133460    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"
Nov 03 13:14:35.133574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.133495    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" event={"ID":"2bf516db-3ff5-4e54-b0f4-2bf041f6b161","Type":"ContainerDied","Data":"d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd"}
Nov 03 13:14:35.133574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.133546    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs" event={"ID":"2bf516db-3ff5-4e54-b0f4-2bf041f6b161","Type":"ContainerDied","Data":"3b1d679b5e1caf2b1c1091422f11ede8413beaf0b249eea5af3fc8b76e60ccd6"}
Nov 03 13:14:35.133574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.133568    4909 scope.go:122] "RemoveContainer" containerID="d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd"
Nov 03 13:14:35.144281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.144266    4909 scope.go:122] "RemoveContainer" containerID="d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd"
Nov 03 13:14:35.144556 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:14:35.144535    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd\": container with ID starting with d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd not found: ID does not exist" containerID="d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd"
Nov 03 13:14:35.144608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.144564    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd"} err="failed to get container status \"d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd\": rpc error: code = NotFound desc = could not find container \"d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd\": container with ID starting with d80b9920fcb3dad63ebaef9842012dfcbd2a1b030415c0c3195431b1957f66fd not found: ID does not exist"
Nov 03 13:14:35.285779 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.285750    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"]
Nov 03 13:14:35.331897 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:35.331868    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dmgvs"]
Nov 03 13:14:36.949378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:36.949331    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845" podUID="d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" containerName="main" containerID="cri-o://fdbc6e69e7832089b8973acf7f088e9a307edffd9b854e83b00f4464ef474f06" gracePeriod=45
Nov 03 13:14:37.127006 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.126375    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" path="/var/lib/kubelet/pods/2bf516db-3ff5-4e54-b0f4-2bf041f6b161/volumes"
Nov 03 13:14:37.149314 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.149282    4909 generic.go:356] "Generic (PLEG): container finished" podID="d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" containerID="fdbc6e69e7832089b8973acf7f088e9a307edffd9b854e83b00f4464ef474f06" exitCode=0
Nov 03 13:14:37.149525 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.149502    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845" event={"ID":"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6","Type":"ContainerDied","Data":"fdbc6e69e7832089b8973acf7f088e9a307edffd9b854e83b00f4464ef474f06"}
Nov 03 13:14:37.208816 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.208758    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:14:37.306249 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306218    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-kube-api-access-5rzbb\" (UniqueName: \"kubernetes.io/projected/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-kube-api-access-5rzbb\") pod \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") "
Nov 03 13:14:37.306404 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306260    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-dshm\") pod \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") "
Nov 03 13:14:37.306404 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306319    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tmp-dir\") pod \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") "
Nov 03 13:14:37.306404 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306343    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-model-cache\") pod \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") "
Nov 03 13:14:37.306404 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306371    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-home\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-home\") pod \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") "
Nov 03 13:14:37.306647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306429    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tls-certs\") pod \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\" (UID: \"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6\") "
Nov 03 13:14:37.306647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306603    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tmp-dir" pod "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" (UID: "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:37.306647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306623    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-model-cache" pod "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" (UID: "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:37.306864 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306839    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-home" pod "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" (UID: "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:37.306988 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306964    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:37.307082 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.306991    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:37.307082 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.307006    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:37.308542 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.308516    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-dshm" pod "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" (UID: "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:37.308632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.308570    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-kube-api-access-5rzbb" pod "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" (UID: "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6"). InnerVolumeSpecName "kube-api-access-5rzbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:37.308632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.308586    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tls-certs" pod "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" (UID: "d3b0c8b9-80ef-45dc-ba0f-8852a040dee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:37.408425 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.408388    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rzbb\" (UniqueName: \"kubernetes.io/projected/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-kube-api-access-5rzbb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:37.408425 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.408419    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:37.408425 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:37.408430    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:38.156511 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:38.156481    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"
Nov 03 13:14:38.156939 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:38.156518    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845" event={"ID":"d3b0c8b9-80ef-45dc-ba0f-8852a040dee6","Type":"ContainerDied","Data":"4ae8d228a58f6d66d2e0b94a8bbde31be6e4de2cc6fc04b319ac4f17c6f34730"}
Nov 03 13:14:38.156939 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:38.156566    4909 scope.go:122] "RemoveContainer" containerID="fdbc6e69e7832089b8973acf7f088e9a307edffd9b854e83b00f4464ef474f06"
Nov 03 13:14:38.308998 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:38.308968    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"]
Nov 03 13:14:38.354914 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:38.354885    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1250b3/rate-test-model-1250b3-kserve-5d8f5d7d65-j9845"]
Nov 03 13:14:39.123838 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:14:39.123808    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" path="/var/lib/kubelet/pods/d3b0c8b9-80ef-45dc-ba0f-8852a040dee6/volumes"
Nov 03 13:14:39.564523 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:14:39.564491    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:15:00.043072 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.043020    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"]
Nov 03 13:15:00.176035 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.175998    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"]
Nov 03 13:15:00.191974 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.191946    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"]
Nov 03 13:15:00.192468 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.192454    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy"
Nov 03 13:15:00.192468 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.192467    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy"
Nov 03 13:15:00.192603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.192515    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" containerName="main"
Nov 03 13:15:00.192603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.192520    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" containerName="main"
Nov 03 13:15:00.192678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.192617    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf516db-3ff5-4e54-b0f4-2bf041f6b161" containerName="istio-proxy"
Nov 03 13:15:00.192678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.192632    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b0c8b9-80ef-45dc-ba0f-8852a040dee6" containerName="main"
Nov 03 13:15:00.192678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.192640    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="283a09fd-6d12-43f2-8672-815f5f4d7863" containerName="istio-proxy"
Nov 03 13:15:00.206838 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.206813    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.207033 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.207012    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"]
Nov 03 13:15:00.210916 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.210895    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"]
Nov 03 13:15:00.211136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.210971    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.215151 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.215128    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.217409 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.217376    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"]
Nov 03 13:15:00.221414 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.221398    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"]
Nov 03 13:15:00.221531 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.221511    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.225123 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.225096    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.282063 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.282029    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141a8a03-06a4-49f2-8137-d49930e9e0ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.282392 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.282364    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb928\" (UniqueName: \"kubernetes.io/projected/2d7c238c-75f3-4f39-b899-a46a57487fcc-kube-api-access-sb928\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.282587 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.282566    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5prt\" (UniqueName: \"kubernetes.io/projected/141a8a03-06a4-49f2-8137-d49930e9e0ba-kube-api-access-n5prt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.283423 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.283385    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c238c-75f3-4f39-b899-a46a57487fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.283633 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.283613    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3957cb6-a26a-45f6-8b93-5abc02b422ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.283890 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.283862    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9tws\" (UniqueName: \"kubernetes.io/projected/536be7e2-613f-49d1-b30a-bb3cd6228494-kube-api-access-d9tws\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.284231 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.283901    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jjf\" (UniqueName: \"kubernetes.io/projected/a3957cb6-a26a-45f6-8b93-5abc02b422ec-kube-api-access-p6jjf\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.284231 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.283951    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d788795-759c-423b-b487-f8e5e44028f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.284231 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.284025    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536be7e2-613f-49d1-b30a-bb3cd6228494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.284231 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.284083    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p295n\" (UniqueName: \"kubernetes.io/projected/2d788795-759c-423b-b487-f8e5e44028f0-kube-api-access-p295n\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.312033 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.311962    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"]
Nov 03 13:15:00.314623 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.314594    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"]
Nov 03 13:15:00.331781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.331763    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:15:00.354269 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.354238    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"]
Nov 03 13:15:00.359456 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.359431    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"]
Nov 03 13:15:00.362319 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.362296    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"]
Nov 03 13:15:00.385077 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385048    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb928\" (UniqueName: \"kubernetes.io/projected/2d7c238c-75f3-4f39-b899-a46a57487fcc-kube-api-access-sb928\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.385173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385101    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5prt\" (UniqueName: \"kubernetes.io/projected/141a8a03-06a4-49f2-8137-d49930e9e0ba-kube-api-access-n5prt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.385173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385134    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c238c-75f3-4f39-b899-a46a57487fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.385261 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385173    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3957cb6-a26a-45f6-8b93-5abc02b422ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.385261 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385198    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9tws\" (UniqueName: \"kubernetes.io/projected/536be7e2-613f-49d1-b30a-bb3cd6228494-kube-api-access-d9tws\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.385261 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385221    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jjf\" (UniqueName: \"kubernetes.io/projected/a3957cb6-a26a-45f6-8b93-5abc02b422ec-kube-api-access-p6jjf\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.385426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385261    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d788795-759c-423b-b487-f8e5e44028f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.385426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385329    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536be7e2-613f-49d1-b30a-bb3cd6228494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.385426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385366    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p295n\" (UniqueName: \"kubernetes.io/projected/2d788795-759c-423b-b487-f8e5e44028f0-kube-api-access-p295n\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.385594 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385456    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141a8a03-06a4-49f2-8137-d49930e9e0ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.386007 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.385984    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c238c-75f3-4f39-b899-a46a57487fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.386086 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.386005    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d788795-759c-423b-b487-f8e5e44028f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.386086 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.386047    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3957cb6-a26a-45f6-8b93-5abc02b422ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.386158 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.386137    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536be7e2-613f-49d1-b30a-bb3cd6228494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.386196 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.386174    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141a8a03-06a4-49f2-8137-d49930e9e0ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.442473 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.442376    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5prt\" (UniqueName: \"kubernetes.io/projected/141a8a03-06a4-49f2-8137-d49930e9e0ba-kube-api-access-n5prt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.442745 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.442549    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb928\" (UniqueName: \"kubernetes.io/projected/2d7c238c-75f3-4f39-b899-a46a57487fcc-kube-api-access-sb928\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.443181 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.443112    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jjf\" (UniqueName: \"kubernetes.io/projected/a3957cb6-a26a-45f6-8b93-5abc02b422ec-kube-api-access-p6jjf\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.445117 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.445092    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9tws\" (UniqueName: \"kubernetes.io/projected/536be7e2-613f-49d1-b30a-bb3cd6228494-kube-api-access-d9tws\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.445407 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.445385    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p295n\" (UniqueName: \"kubernetes.io/projected/2d788795-759c-423b-b487-f8e5e44028f0-kube-api-access-p295n\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.524009 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.523973    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:00.532816 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.532790    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:00.539889 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.539861    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:00.547851 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.547822    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:00.556038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.556005    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:00.943989 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.943961    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"]
Nov 03 13:15:00.945491 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:15:00.945468    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3957cb6_a26a_45f6_8b93_5abc02b422ec.slice/crio-51e002c6d80c01bd84b1eeb539aa2d2cd335d41540b80c286d40788c0c2e1c17 WatchSource:0}: Error finding container 51e002c6d80c01bd84b1eeb539aa2d2cd335d41540b80c286d40788c0c2e1c17: Status 404 returned error can't find the container with id 51e002c6d80c01bd84b1eeb539aa2d2cd335d41540b80c286d40788c0c2e1c17
Nov 03 13:15:00.997039 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:00.997011    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"]
Nov 03 13:15:00.997902 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:15:00.997872    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7c238c_75f3_4f39_b899_a46a57487fcc.slice/crio-1d133306de713b3794019c5d790d829ce65085da34754e29bd987560073fd980 WatchSource:0}: Error finding container 1d133306de713b3794019c5d790d829ce65085da34754e29bd987560073fd980: Status 404 returned error can't find the container with id 1d133306de713b3794019c5d790d829ce65085da34754e29bd987560073fd980
Nov 03 13:15:00.999355 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:15:00.999300    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141a8a03_06a4_49f2_8137_d49930e9e0ba.slice/crio-7e7c87e5ad1f1cfc9554ca32b5f0bd2c7401ba11313d63fcf4aaef63ed0485ba WatchSource:0}: Error finding container 7e7c87e5ad1f1cfc9554ca32b5f0bd2c7401ba11313d63fcf4aaef63ed0485ba: Status 404 returned error can't find the container with id 7e7c87e5ad1f1cfc9554ca32b5f0bd2c7401ba11313d63fcf4aaef63ed0485ba
Nov 03 13:15:01.002367 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:15:01.002343    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536be7e2_613f_49d1_b30a_bb3cd6228494.slice/crio-d3b2d1de32476825bb4e0d834fe6bcf18ed45047a2f3de0c78d85719a38f451d WatchSource:0}: Error finding container d3b2d1de32476825bb4e0d834fe6bcf18ed45047a2f3de0c78d85719a38f451d: Status 404 returned error can't find the container with id d3b2d1de32476825bb4e0d834fe6bcf18ed45047a2f3de0c78d85719a38f451d
Nov 03 13:15:01.002509 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.002493    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"]
Nov 03 13:15:01.005644 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.005619    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"]
Nov 03 13:15:01.055217 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.055190    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"]
Nov 03 13:15:01.141736 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:15:01.141688    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d788795_759c_423b_b487_f8e5e44028f0.slice/crio-3388b9f94cc006b8bc1b861cb11604dc560dc81f253984154682b0e4da696aae WatchSource:0}: Error finding container 3388b9f94cc006b8bc1b861cb11604dc560dc81f253984154682b0e4da696aae: Status 404 returned error can't find the container with id 3388b9f94cc006b8bc1b861cb11604dc560dc81f253984154682b0e4da696aae
Nov 03 13:15:01.318915 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.318883    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" event={"ID":"a3957cb6-a26a-45f6-8b93-5abc02b422ec","Type":"ContainerStarted","Data":"51e002c6d80c01bd84b1eeb539aa2d2cd335d41540b80c286d40788c0c2e1c17"}
Nov 03 13:15:01.320151 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.320123    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" event={"ID":"536be7e2-613f-49d1-b30a-bb3cd6228494","Type":"ContainerStarted","Data":"d3b2d1de32476825bb4e0d834fe6bcf18ed45047a2f3de0c78d85719a38f451d"}
Nov 03 13:15:01.321358 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.321337    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" event={"ID":"2d7c238c-75f3-4f39-b899-a46a57487fcc","Type":"ContainerStarted","Data":"1d133306de713b3794019c5d790d829ce65085da34754e29bd987560073fd980"}
Nov 03 13:15:01.322543 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.322521    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" event={"ID":"141a8a03-06a4-49f2-8137-d49930e9e0ba","Type":"ContainerStarted","Data":"7e7c87e5ad1f1cfc9554ca32b5f0bd2c7401ba11313d63fcf4aaef63ed0485ba"}
Nov 03 13:15:01.323552 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:01.323531    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" event={"ID":"2d788795-759c-423b-b487-f8e5e44028f0","Type":"ContainerStarted","Data":"3388b9f94cc006b8bc1b861cb11604dc560dc81f253984154682b0e4da696aae"}
Nov 03 13:15:02.330614 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.330572    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d788795-759c-423b-b487-f8e5e44028f0" containerID="1a3e2ab090f5b478a9ce6f0568a7245e1a14e22c8bb9f5ccf8eb351cefa1022b" exitCode=60
Nov 03 13:15:02.330614 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.330604    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" event={"ID":"2d788795-759c-423b-b487-f8e5e44028f0","Type":"ContainerDied","Data":"1a3e2ab090f5b478a9ce6f0568a7245e1a14e22c8bb9f5ccf8eb351cefa1022b"}
Nov 03 13:15:02.331143 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.330954    4909 scope.go:122] "RemoveContainer" containerID="1a3e2ab090f5b478a9ce6f0568a7245e1a14e22c8bb9f5ccf8eb351cefa1022b"
Nov 03 13:15:02.332593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.332434    4909 generic.go:356] "Generic (PLEG): container finished" podID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerID="be638d5e95ca6122ed736505c97e7dbf5e452bfbb4941b712b10204d2259881d" exitCode=60
Nov 03 13:15:02.332593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.332503    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" event={"ID":"a3957cb6-a26a-45f6-8b93-5abc02b422ec","Type":"ContainerDied","Data":"be638d5e95ca6122ed736505c97e7dbf5e452bfbb4941b712b10204d2259881d"}
Nov 03 13:15:02.332866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.332689    4909 scope.go:122] "RemoveContainer" containerID="be638d5e95ca6122ed736505c97e7dbf5e452bfbb4941b712b10204d2259881d"
Nov 03 13:15:02.334393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.334372    4909 generic.go:356] "Generic (PLEG): container finished" podID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerID="b28c5a2451a3770012d350d420c78422b64c93190d12061237e2b5d3d9177e34" exitCode=60
Nov 03 13:15:02.334557 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.334521    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" event={"ID":"536be7e2-613f-49d1-b30a-bb3cd6228494","Type":"ContainerDied","Data":"b28c5a2451a3770012d350d420c78422b64c93190d12061237e2b5d3d9177e34"}
Nov 03 13:15:02.334776 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.334759    4909 scope.go:122] "RemoveContainer" containerID="b28c5a2451a3770012d350d420c78422b64c93190d12061237e2b5d3d9177e34"
Nov 03 13:15:02.336895 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.336876    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerID="df81e71caf8219435fea685ff3af82a31d3f92c0c86f33b2eeaebf657c42471f" exitCode=60
Nov 03 13:15:02.337001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.336941    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" event={"ID":"2d7c238c-75f3-4f39-b899-a46a57487fcc","Type":"ContainerDied","Data":"df81e71caf8219435fea685ff3af82a31d3f92c0c86f33b2eeaebf657c42471f"}
Nov 03 13:15:02.337109 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.337093    4909 scope.go:122] "RemoveContainer" containerID="df81e71caf8219435fea685ff3af82a31d3f92c0c86f33b2eeaebf657c42471f"
Nov 03 13:15:02.338465 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.338443    4909 generic.go:356] "Generic (PLEG): container finished" podID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerID="d35803177e7fafa50b775170128d270eafbb936a07cdf87a812162fe5bd3e7e2" exitCode=60
Nov 03 13:15:02.338571 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.338469    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" event={"ID":"141a8a03-06a4-49f2-8137-d49930e9e0ba","Type":"ContainerDied","Data":"d35803177e7fafa50b775170128d270eafbb936a07cdf87a812162fe5bd3e7e2"}
Nov 03 13:15:02.338655 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:02.338641    4909 scope.go:122] "RemoveContainer" containerID="d35803177e7fafa50b775170128d270eafbb936a07cdf87a812162fe5bd3e7e2"
Nov 03 13:15:03.349030 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.348994    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" event={"ID":"2d7c238c-75f3-4f39-b899-a46a57487fcc","Type":"ContainerStarted","Data":"011cfd0a14ca540a57df57c497f0f28f3a36a5786c7c20753c15cfb560a5aff7"}
Nov 03 13:15:03.351245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.351219    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" event={"ID":"141a8a03-06a4-49f2-8137-d49930e9e0ba","Type":"ContainerStarted","Data":"610e8e05c18380c425eeca4418b67a439572cc87936081d8fec4f83ecf3b9fdb"}
Nov 03 13:15:03.353110 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.353083    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" event={"ID":"2d788795-759c-423b-b487-f8e5e44028f0","Type":"ContainerStarted","Data":"fd59f81a3f6162a172beeb2fade61cbde3ded8b27a1a1a5e9c74d0e37c797542"}
Nov 03 13:15:03.354974 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.354950    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" event={"ID":"a3957cb6-a26a-45f6-8b93-5abc02b422ec","Type":"ContainerStarted","Data":"199b64ad210ac6fa2c354cd180ebbc1e2e8b06a3ba21cf1769055882aa263562"}
Nov 03 13:15:03.356942 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.356920    4909 generic.go:356] "Generic (PLEG): container finished" podID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerID="444de1c96f9020e7b579a988393ba092ff94fd086f25806b79c2287ea183b97f" exitCode=60
Nov 03 13:15:03.357043 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.356987    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" event={"ID":"536be7e2-613f-49d1-b30a-bb3cd6228494","Type":"ContainerDied","Data":"444de1c96f9020e7b579a988393ba092ff94fd086f25806b79c2287ea183b97f"}
Nov 03 13:15:03.357116 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.357040    4909 scope.go:122] "RemoveContainer" containerID="b28c5a2451a3770012d350d420c78422b64c93190d12061237e2b5d3d9177e34"
Nov 03 13:15:03.357179 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.357133    4909 scope.go:122] "RemoveContainer" containerID="444de1c96f9020e7b579a988393ba092ff94fd086f25806b79c2287ea183b97f"
Nov 03 13:15:03.357397 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:03.357378    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs_odh-ai-gateway-infra(536be7e2-613f-49d1-b30a-bb3cd6228494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494"
Nov 03 13:15:03.455239 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.455191    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" podStartSLOduration=3.45517828 podStartE2EDuration="3.45517828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:03.453837074 +0000 UTC m=+5052.466303738" watchObservedRunningTime="2026-11-03 13:15:03.45517828 +0000 UTC m=+5052.467644942"
Nov 03 13:15:03.670237 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.670186    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" podStartSLOduration=3.670169944 podStartE2EDuration="3.670169944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:03.667959101 +0000 UTC m=+5052.680425763" watchObservedRunningTime="2026-11-03 13:15:03.670169944 +0000 UTC m=+5052.682636607"
Nov 03 13:15:03.777917 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.777865    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" podStartSLOduration=3.77785159 podStartE2EDuration="3.77785159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:03.775132535 +0000 UTC m=+5052.787599200" watchObservedRunningTime="2026-11-03 13:15:03.77785159 +0000 UTC m=+5052.790318252"
Nov 03 13:15:03.882791 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:03.882592    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" podStartSLOduration=3.882575675 podStartE2EDuration="3.882575675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:03.881855486 +0000 UTC m=+5052.894322148" watchObservedRunningTime="2026-11-03 13:15:03.882575675 +0000 UTC m=+5052.895042341"
Nov 03 13:15:04.365516 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.365478    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerID="011cfd0a14ca540a57df57c497f0f28f3a36a5786c7c20753c15cfb560a5aff7" exitCode=60
Nov 03 13:15:04.366023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.365519    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" event={"ID":"2d7c238c-75f3-4f39-b899-a46a57487fcc","Type":"ContainerDied","Data":"011cfd0a14ca540a57df57c497f0f28f3a36a5786c7c20753c15cfb560a5aff7"}
Nov 03 13:15:04.366023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.365586    4909 scope.go:122] "RemoveContainer" containerID="df81e71caf8219435fea685ff3af82a31d3f92c0c86f33b2eeaebf657c42471f"
Nov 03 13:15:04.366023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.365830    4909 scope.go:122] "RemoveContainer" containerID="011cfd0a14ca540a57df57c497f0f28f3a36a5786c7c20753c15cfb560a5aff7"
Nov 03 13:15:04.366175 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:04.366103    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26_odh-ai-gateway-infra(2d7c238c-75f3-4f39-b899-a46a57487fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc"
Nov 03 13:15:04.367877 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.367843    4909 generic.go:356] "Generic (PLEG): container finished" podID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerID="610e8e05c18380c425eeca4418b67a439572cc87936081d8fec4f83ecf3b9fdb" exitCode=60
Nov 03 13:15:04.367995 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.367946    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" event={"ID":"141a8a03-06a4-49f2-8137-d49930e9e0ba","Type":"ContainerDied","Data":"610e8e05c18380c425eeca4418b67a439572cc87936081d8fec4f83ecf3b9fdb"}
Nov 03 13:15:04.368132 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.368111    4909 scope.go:122] "RemoveContainer" containerID="610e8e05c18380c425eeca4418b67a439572cc87936081d8fec4f83ecf3b9fdb"
Nov 03 13:15:04.368528 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:04.368331    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt_odh-ai-gateway-infra(141a8a03-06a4-49f2-8137-d49930e9e0ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba"
Nov 03 13:15:04.369956 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.369932    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d788795-759c-423b-b487-f8e5e44028f0" containerID="fd59f81a3f6162a172beeb2fade61cbde3ded8b27a1a1a5e9c74d0e37c797542" exitCode=60
Nov 03 13:15:04.370064 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.369967    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" event={"ID":"2d788795-759c-423b-b487-f8e5e44028f0","Type":"ContainerDied","Data":"fd59f81a3f6162a172beeb2fade61cbde3ded8b27a1a1a5e9c74d0e37c797542"}
Nov 03 13:15:04.370218 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.370204    4909 scope.go:122] "RemoveContainer" containerID="fd59f81a3f6162a172beeb2fade61cbde3ded8b27a1a1a5e9c74d0e37c797542"
Nov 03 13:15:04.370429 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:04.370408    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw_odh-ai-gateway-infra(2d788795-759c-423b-b487-f8e5e44028f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" podUID="2d788795-759c-423b-b487-f8e5e44028f0"
Nov 03 13:15:04.372038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.372022    4909 generic.go:356] "Generic (PLEG): container finished" podID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerID="199b64ad210ac6fa2c354cd180ebbc1e2e8b06a3ba21cf1769055882aa263562" exitCode=60
Nov 03 13:15:04.372140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.372103    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" event={"ID":"a3957cb6-a26a-45f6-8b93-5abc02b422ec","Type":"ContainerDied","Data":"199b64ad210ac6fa2c354cd180ebbc1e2e8b06a3ba21cf1769055882aa263562"}
Nov 03 13:15:04.372286 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.372269    4909 scope.go:122] "RemoveContainer" containerID="199b64ad210ac6fa2c354cd180ebbc1e2e8b06a3ba21cf1769055882aa263562"
Nov 03 13:15:04.372460 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:04.372443    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8_odh-ai-gateway-infra(a3957cb6-a26a-45f6-8b93-5abc02b422ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec"
Nov 03 13:15:04.374009 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.373990    4909 scope.go:122] "RemoveContainer" containerID="444de1c96f9020e7b579a988393ba092ff94fd086f25806b79c2287ea183b97f"
Nov 03 13:15:04.374231 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:04.374207    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs_odh-ai-gateway-infra(536be7e2-613f-49d1-b30a-bb3cd6228494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494"
Nov 03 13:15:04.381917 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.381885    4909 scope.go:122] "RemoveContainer" containerID="d35803177e7fafa50b775170128d270eafbb936a07cdf87a812162fe5bd3e7e2"
Nov 03 13:15:04.394579 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.394561    4909 scope.go:122] "RemoveContainer" containerID="1a3e2ab090f5b478a9ce6f0568a7245e1a14e22c8bb9f5ccf8eb351cefa1022b"
Nov 03 13:15:04.406584 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:04.406566    4909 scope.go:122] "RemoveContainer" containerID="be638d5e95ca6122ed736505c97e7dbf5e452bfbb4941b712b10204d2259881d"
Nov 03 13:15:05.044347 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.044307    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" podUID="ded63b27-8f76-41d1-a992-38982f6a69b4" containerName="payload-processing" containerID="cri-o://bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc" gracePeriod=25
Nov 03 13:15:05.177034 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.176993    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" podUID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" containerName="payload-pre-processing" containerID="cri-o://8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962" gracePeriod=25
Nov 03 13:15:05.381624 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.381539    4909 scope.go:122] "RemoveContainer" containerID="011cfd0a14ca540a57df57c497f0f28f3a36a5786c7c20753c15cfb560a5aff7"
Nov 03 13:15:05.382118 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:05.381807    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26_odh-ai-gateway-infra(2d7c238c-75f3-4f39-b899-a46a57487fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc"
Nov 03 13:15:05.383168 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.383150    4909 scope.go:122] "RemoveContainer" containerID="610e8e05c18380c425eeca4418b67a439572cc87936081d8fec4f83ecf3b9fdb"
Nov 03 13:15:05.383376 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:05.383345    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt_odh-ai-gateway-infra(141a8a03-06a4-49f2-8137-d49930e9e0ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba"
Nov 03 13:15:05.384682 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.384662    4909 scope.go:122] "RemoveContainer" containerID="fd59f81a3f6162a172beeb2fade61cbde3ded8b27a1a1a5e9c74d0e37c797542"
Nov 03 13:15:05.384902 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:05.384882    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw_odh-ai-gateway-infra(2d788795-759c-423b-b487-f8e5e44028f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" podUID="2d788795-759c-423b-b487-f8e5e44028f0"
Nov 03 13:15:05.386063 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.386047    4909 scope.go:122] "RemoveContainer" containerID="199b64ad210ac6fa2c354cd180ebbc1e2e8b06a3ba21cf1769055882aa263562"
Nov 03 13:15:05.386227 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:05.386210    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8_odh-ai-gateway-infra(a3957cb6-a26a-45f6-8b93-5abc02b422ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec"
Nov 03 13:15:05.806513 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.806480    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"]
Nov 03 13:15:05.821367 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.821315    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.867756 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.867725    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"]
Nov 03 13:15:05.869802 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.869751    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.869944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.869818    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.869944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.869886    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9h5t\" (UniqueName: \"kubernetes.io/projected/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-kube-api-access-z9h5t\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.869944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.869923    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:15:05.970631 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.970594    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9h5t\" (UniqueName: \"kubernetes.io/projected/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-kube-api-access-z9h5t\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.970839 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.970820    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.970912 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.970869    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.971534 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.971513    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:05.973251 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:05.973230    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:06.025655 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:06.025621    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9h5t\" (UniqueName: \"kubernetes.io/projected/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-kube-api-access-z9h5t\") pod \"maas-api-revoke-sub-fedeacaf-rhqqq\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:06.134915 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:06.134855    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:06.530522 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:06.530485    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"]
Nov 03 13:15:06.531192 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:15:06.531163    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b74642_9783_4509_8eaa_0b2c33f0f8e2.slice/crio-ec3c57341e82394b24ab522a47e00e4aa273c9a3323006e5fc290b5c1eeae04f WatchSource:0}: Error finding container ec3c57341e82394b24ab522a47e00e4aa273c9a3323006e5fc290b5c1eeae04f: Status 404 returned error can't find the container with id ec3c57341e82394b24ab522a47e00e4aa273c9a3323006e5fc290b5c1eeae04f
Nov 03 13:15:07.401595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:07.401558    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq" event={"ID":"b4b74642-9783-4509-8eaa-0b2c33f0f8e2","Type":"ContainerStarted","Data":"23ee7c0eba8ee84d2d02ec192f62eac1d36d7e4488e0c535de31758ed1fb0c6d"}
Nov 03 13:15:07.401791 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:07.401609    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq" event={"ID":"b4b74642-9783-4509-8eaa-0b2c33f0f8e2","Type":"ContainerStarted","Data":"ec3c57341e82394b24ab522a47e00e4aa273c9a3323006e5fc290b5c1eeae04f"}
Nov 03 13:15:07.513882 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:07.513826    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq" podStartSLOduration=2.513808124 podStartE2EDuration="2.513808124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:07.512177459 +0000 UTC m=+5056.524644113" watchObservedRunningTime="2026-11-03 13:15:07.513808124 +0000 UTC m=+5056.526274787"
Nov 03 13:15:08.408799 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:08.408765    4909 generic.go:356] "Generic (PLEG): container finished" podID="b4b74642-9783-4509-8eaa-0b2c33f0f8e2" containerID="23ee7c0eba8ee84d2d02ec192f62eac1d36d7e4488e0c535de31758ed1fb0c6d" exitCode=0
Nov 03 13:15:08.409217 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:08.408847    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq" event={"ID":"b4b74642-9783-4509-8eaa-0b2c33f0f8e2","Type":"ContainerDied","Data":"23ee7c0eba8ee84d2d02ec192f62eac1d36d7e4488e0c535de31758ed1fb0c6d"}
Nov 03 13:15:09.564896 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:09.564862    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:15:09.565735 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.565718    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:09.610773 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.610750    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-xks-ca\") pod \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") "
Nov 03 13:15:09.610909 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.610823    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-openshift-ca\") pod \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") "
Nov 03 13:15:09.610909 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.610906    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-kube-api-access-z9h5t\" (UniqueName: \"kubernetes.io/projected/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-kube-api-access-z9h5t\") pod \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\" (UID: \"b4b74642-9783-4509-8eaa-0b2c33f0f8e2\") "
Nov 03 13:15:09.611185 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.611157    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-openshift-ca" pod "b4b74642-9783-4509-8eaa-0b2c33f0f8e2" (UID: "b4b74642-9783-4509-8eaa-0b2c33f0f8e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:09.611347 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.611334    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-openshift-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:09.612822 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.612795    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-xks-ca" pod "b4b74642-9783-4509-8eaa-0b2c33f0f8e2" (UID: "b4b74642-9783-4509-8eaa-0b2c33f0f8e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:09.612989 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.612968    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-kube-api-access-z9h5t" pod "b4b74642-9783-4509-8eaa-0b2c33f0f8e2" (UID: "b4b74642-9783-4509-8eaa-0b2c33f0f8e2"). InnerVolumeSpecName "kube-api-access-z9h5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:09.695803 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.695769    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" podUID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:09.712833 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.712802    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-maas-api-xks-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:09.712965 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:09.712835    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9h5t\" (UniqueName: \"kubernetes.io/projected/b4b74642-9783-4509-8eaa-0b2c33f0f8e2-kube-api-access-z9h5t\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:10.422575 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:10.422543    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"
Nov 03 13:15:10.422767 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:10.422571    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq" event={"ID":"b4b74642-9783-4509-8eaa-0b2c33f0f8e2","Type":"ContainerDied","Data":"ec3c57341e82394b24ab522a47e00e4aa273c9a3323006e5fc290b5c1eeae04f"}
Nov 03 13:15:10.422767 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:10.422642    4909 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec3c57341e82394b24ab522a47e00e4aa273c9a3323006e5fc290b5c1eeae04f"
Nov 03 13:15:10.705920 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:10.705886    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" podUID="ded63b27-8f76-41d1-a992-38982f6a69b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:11.108294 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:11.108206    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"]
Nov 03 13:15:11.271372 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:11.271336    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"]
Nov 03 13:15:14.116602 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:14.116574    4909 scope.go:122] "RemoveContainer" containerID="610e8e05c18380c425eeca4418b67a439572cc87936081d8fec4f83ecf3b9fdb"
Nov 03 13:15:15.116870 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:15.116843    4909 scope.go:122] "RemoveContainer" containerID="199b64ad210ac6fa2c354cd180ebbc1e2e8b06a3ba21cf1769055882aa263562"
Nov 03 13:15:15.117260 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:15.116989    4909 scope.go:122] "RemoveContainer" containerID="444de1c96f9020e7b579a988393ba092ff94fd086f25806b79c2287ea183b97f"
Nov 03 13:15:15.458485 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:15.458458    4909 generic.go:356] "Generic (PLEG): container finished" podID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerID="21ba24b38b800a897b0a0dca099145d21892a2a2d08401d3fea08727bc0a55d4" exitCode=60
Nov 03 13:15:15.458606 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:15.458528    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" event={"ID":"141a8a03-06a4-49f2-8137-d49930e9e0ba","Type":"ContainerDied","Data":"21ba24b38b800a897b0a0dca099145d21892a2a2d08401d3fea08727bc0a55d4"}
Nov 03 13:15:15.458606 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:15.458594    4909 scope.go:122] "RemoveContainer" containerID="610e8e05c18380c425eeca4418b67a439572cc87936081d8fec4f83ecf3b9fdb"
Nov 03 13:15:15.458992 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:15.458968    4909 scope.go:122] "RemoveContainer" containerID="21ba24b38b800a897b0a0dca099145d21892a2a2d08401d3fea08727bc0a55d4"
Nov 03 13:15:15.459409 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:15.459381    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt_odh-ai-gateway-infra(141a8a03-06a4-49f2-8137-d49930e9e0ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba"
Nov 03 13:15:16.109157 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.109118    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" podUID="4958a745-d677-4897-8d31-cf29f2a67b9c" containerName="payload-processing" containerID="cri-o://39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080" gracePeriod=25
Nov 03 13:15:16.116099 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.116071    4909 scope.go:122] "RemoveContainer" containerID="fd59f81a3f6162a172beeb2fade61cbde3ded8b27a1a1a5e9c74d0e37c797542"
Nov 03 13:15:16.272007 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.271936    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" podUID="cc822e2b-def1-4eb9-a22b-f03dc395205c" containerName="payload-pre-processing" containerID="cri-o://e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2" gracePeriod=25
Nov 03 13:15:16.468569 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.468545    4909 generic.go:356] "Generic (PLEG): container finished" podID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerID="f43ffd7a11876740a74bbddf93609ec76efb2ddc3cc45033b88bed8e153d038a" exitCode=60
Nov 03 13:15:16.468688 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.468618    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" event={"ID":"536be7e2-613f-49d1-b30a-bb3cd6228494","Type":"ContainerDied","Data":"f43ffd7a11876740a74bbddf93609ec76efb2ddc3cc45033b88bed8e153d038a"}
Nov 03 13:15:16.468688 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.468684    4909 scope.go:122] "RemoveContainer" containerID="444de1c96f9020e7b579a988393ba092ff94fd086f25806b79c2287ea183b97f"
Nov 03 13:15:16.469017 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.469003    4909 scope.go:122] "RemoveContainer" containerID="f43ffd7a11876740a74bbddf93609ec76efb2ddc3cc45033b88bed8e153d038a"
Nov 03 13:15:16.469307 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:16.469287    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs_odh-ai-gateway-infra(536be7e2-613f-49d1-b30a-bb3cd6228494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494"
Nov 03 13:15:16.472061 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.472040    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" event={"ID":"2d788795-759c-423b-b487-f8e5e44028f0","Type":"ContainerStarted","Data":"38f396b37f7bc722e6718bbb280beaea8adfa168ddb36c65899dac071a542f41"}
Nov 03 13:15:16.473868 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.473848    4909 generic.go:356] "Generic (PLEG): container finished" podID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerID="31c62ff73a01601952c76c7f25d97ce00acf60faaef77cd08c1dbf668f06fc85" exitCode=60
Nov 03 13:15:16.473975 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.473920    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" event={"ID":"a3957cb6-a26a-45f6-8b93-5abc02b422ec","Type":"ContainerDied","Data":"31c62ff73a01601952c76c7f25d97ce00acf60faaef77cd08c1dbf668f06fc85"}
Nov 03 13:15:16.474195 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.474181    4909 scope.go:122] "RemoveContainer" containerID="31c62ff73a01601952c76c7f25d97ce00acf60faaef77cd08c1dbf668f06fc85"
Nov 03 13:15:16.474411 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:16.474393    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8_odh-ai-gateway-infra(a3957cb6-a26a-45f6-8b93-5abc02b422ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec"
Nov 03 13:15:16.480840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.480818    4909 scope.go:122] "RemoveContainer" containerID="199b64ad210ac6fa2c354cd180ebbc1e2e8b06a3ba21cf1769055882aa263562"
Nov 03 13:15:16.600864 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.600788    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"]
Nov 03 13:15:16.631208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.631172    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"]
Nov 03 13:15:16.631996 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.631966    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b4b74642-9783-4509-8eaa-0b2c33f0f8e2" containerName="revoke-keys"
Nov 03 13:15:16.631996 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.631994    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b74642-9783-4509-8eaa-0b2c33f0f8e2" containerName="revoke-keys"
Nov 03 13:15:16.632317 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.632297    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b74642-9783-4509-8eaa-0b2c33f0f8e2" containerName="revoke-keys"
Nov 03 13:15:16.638160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.638134    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.689500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.687883    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:15:16.703208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.703180    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"]
Nov 03 13:15:16.764485 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.764457    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:16.811646 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.811617    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.811813 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.811649    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.811813 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.811762    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccjg6\" (UniqueName: \"kubernetes.io/projected/ef3638d1-389f-41b8-8fa1-0d3053a39f63-kube-api-access-ccjg6\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.913017 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.912931    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141a8a03-06a4-49f2-8137-d49930e9e0ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141a8a03-06a4-49f2-8137-d49930e9e0ba-maas-api-service-ca\") pod \"141a8a03-06a4-49f2-8137-d49930e9e0ba\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") "
Nov 03 13:15:16.913017 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.912972    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141a8a03-06a4-49f2-8137-d49930e9e0ba-kube-api-access-n5prt\" (UniqueName: \"kubernetes.io/projected/141a8a03-06a4-49f2-8137-d49930e9e0ba-kube-api-access-n5prt\") pod \"141a8a03-06a4-49f2-8137-d49930e9e0ba\" (UID: \"141a8a03-06a4-49f2-8137-d49930e9e0ba\") "
Nov 03 13:15:16.913262 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.913220    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccjg6\" (UniqueName: \"kubernetes.io/projected/ef3638d1-389f-41b8-8fa1-0d3053a39f63-kube-api-access-ccjg6\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.913381 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.913355    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a8a03-06a4-49f2-8137-d49930e9e0ba-maas-api-service-ca" pod "141a8a03-06a4-49f2-8137-d49930e9e0ba" (UID: "141a8a03-06a4-49f2-8137-d49930e9e0ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:16.913519 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.913502    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.913601 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.913532    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.913712 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.913680    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141a8a03-06a4-49f2-8137-d49930e9e0ba-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:16.914558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.914535    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.916113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.916090    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:16.916411 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.916389    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a8a03-06a4-49f2-8137-d49930e9e0ba-kube-api-access-n5prt" pod "141a8a03-06a4-49f2-8137-d49930e9e0ba" (UID: "141a8a03-06a4-49f2-8137-d49930e9e0ba"). InnerVolumeSpecName "kube-api-access-n5prt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:16.966635 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:16.966615    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccjg6\" (UniqueName: \"kubernetes.io/projected/ef3638d1-389f-41b8-8fa1-0d3053a39f63-kube-api-access-ccjg6\") pod \"maas-api-revoke-sub-8be06a04-6gdzr\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:17.014397 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.014371    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5prt\" (UniqueName: \"kubernetes.io/projected/141a8a03-06a4-49f2-8137-d49930e9e0ba-kube-api-access-n5prt\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:17.116117 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.116089    4909 scope.go:122] "RemoveContainer" containerID="011cfd0a14ca540a57df57c497f0f28f3a36a5786c7c20753c15cfb560a5aff7"
Nov 03 13:15:17.253994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.253972    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:17.483509 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.483462    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" event={"ID":"2d7c238c-75f3-4f39-b899-a46a57487fcc","Type":"ContainerStarted","Data":"2725736b5b5d3e658689ce93088013ed5d9834d55187cecd36474d4216105650"}
Nov 03 13:15:17.484821 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.484802    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"
Nov 03 13:15:17.484948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.484826    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt" event={"ID":"141a8a03-06a4-49f2-8137-d49930e9e0ba","Type":"ContainerDied","Data":"7e7c87e5ad1f1cfc9554ca32b5f0bd2c7401ba11313d63fcf4aaef63ed0485ba"}
Nov 03 13:15:17.484948 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.484884    4909 scope.go:122] "RemoveContainer" containerID="21ba24b38b800a897b0a0dca099145d21892a2a2d08401d3fea08727bc0a55d4"
Nov 03 13:15:17.486807 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.486784    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d788795-759c-423b-b487-f8e5e44028f0" containerID="38f396b37f7bc722e6718bbb280beaea8adfa168ddb36c65899dac071a542f41" exitCode=60
Nov 03 13:15:17.486938 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.486888    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" event={"ID":"2d788795-759c-423b-b487-f8e5e44028f0","Type":"ContainerDied","Data":"38f396b37f7bc722e6718bbb280beaea8adfa168ddb36c65899dac071a542f41"}
Nov 03 13:15:17.487253 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.487229    4909 scope.go:122] "RemoveContainer" containerID="38f396b37f7bc722e6718bbb280beaea8adfa168ddb36c65899dac071a542f41"
Nov 03 13:15:17.487531 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:17.487505    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw_odh-ai-gateway-infra(2d788795-759c-423b-b487-f8e5e44028f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" podUID="2d788795-759c-423b-b487-f8e5e44028f0"
Nov 03 13:15:17.493876 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.493858    4909 scope.go:122] "RemoveContainer" containerID="fd59f81a3f6162a172beeb2fade61cbde3ded8b27a1a1a5e9c74d0e37c797542"
Nov 03 13:15:17.588845 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.588816    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"]
Nov 03 13:15:17.634724 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.634668    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"]
Nov 03 13:15:17.662582 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:15:17.662552    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3638d1_389f_41b8_8fa1_0d3053a39f63.slice/crio-51a19935915b36c8938fb556d7a177c8d92f416e2f26ac94a46050cb2cc40628 WatchSource:0}: Error finding container 51a19935915b36c8938fb556d7a177c8d92f416e2f26ac94a46050cb2cc40628: Status 404 returned error can't find the container with id 51a19935915b36c8938fb556d7a177c8d92f416e2f26ac94a46050cb2cc40628
Nov 03 13:15:17.697810 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.697783    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"]
Nov 03 13:15:17.739874 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.739845    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:17.770845 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.770813    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:15:17.804142 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.804104    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"]
Nov 03 13:15:17.825289 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.825263    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536be7e2-613f-49d1-b30a-bb3cd6228494-kube-api-access-d9tws\" (UniqueName: \"kubernetes.io/projected/536be7e2-613f-49d1-b30a-bb3cd6228494-kube-api-access-d9tws\") pod \"536be7e2-613f-49d1-b30a-bb3cd6228494\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") "
Nov 03 13:15:17.825544 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.825525    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536be7e2-613f-49d1-b30a-bb3cd6228494-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536be7e2-613f-49d1-b30a-bb3cd6228494-maas-api-service-ca\") pod \"536be7e2-613f-49d1-b30a-bb3cd6228494\" (UID: \"536be7e2-613f-49d1-b30a-bb3cd6228494\") "
Nov 03 13:15:17.825897 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.825867    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536be7e2-613f-49d1-b30a-bb3cd6228494-maas-api-service-ca" pod "536be7e2-613f-49d1-b30a-bb3cd6228494" (UID: "536be7e2-613f-49d1-b30a-bb3cd6228494"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:17.826024 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.826008    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536be7e2-613f-49d1-b30a-bb3cd6228494-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:17.827774 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.827752    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536be7e2-613f-49d1-b30a-bb3cd6228494-kube-api-access-d9tws" pod "536be7e2-613f-49d1-b30a-bb3cd6228494" (UID: "536be7e2-613f-49d1-b30a-bb3cd6228494"). InnerVolumeSpecName "kube-api-access-d9tws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:17.880493 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.880473    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:17.927611 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.927582    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9tws\" (UniqueName: \"kubernetes.io/projected/536be7e2-613f-49d1-b30a-bb3cd6228494-kube-api-access-d9tws\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:17.951666 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.951640    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"]
Nov 03 13:15:17.975309 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:17.975286    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895195-c9pqt"]
Nov 03 13:15:18.028005 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.027940    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3957cb6-a26a-45f6-8b93-5abc02b422ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3957cb6-a26a-45f6-8b93-5abc02b422ec-maas-api-service-ca\") pod \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") "
Nov 03 13:15:18.028128 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.028116    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3957cb6-a26a-45f6-8b93-5abc02b422ec-kube-api-access-p6jjf\" (UniqueName: \"kubernetes.io/projected/a3957cb6-a26a-45f6-8b93-5abc02b422ec-kube-api-access-p6jjf\") pod \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\" (UID: \"a3957cb6-a26a-45f6-8b93-5abc02b422ec\") "
Nov 03 13:15:18.028299 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.028279    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3957cb6-a26a-45f6-8b93-5abc02b422ec-maas-api-service-ca" pod "a3957cb6-a26a-45f6-8b93-5abc02b422ec" (UID: "a3957cb6-a26a-45f6-8b93-5abc02b422ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:18.028500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.028479    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3957cb6-a26a-45f6-8b93-5abc02b422ec-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:18.030156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.030132    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3957cb6-a26a-45f6-8b93-5abc02b422ec-kube-api-access-p6jjf" pod "a3957cb6-a26a-45f6-8b93-5abc02b422ec" (UID: "a3957cb6-a26a-45f6-8b93-5abc02b422ec"). InnerVolumeSpecName "kube-api-access-p6jjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:18.129302 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.129271    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jjf\" (UniqueName: \"kubernetes.io/projected/a3957cb6-a26a-45f6-8b93-5abc02b422ec-kube-api-access-p6jjf\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:18.495367 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.495328    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"
Nov 03 13:15:18.495367 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.495354    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs" event={"ID":"536be7e2-613f-49d1-b30a-bb3cd6228494","Type":"ContainerDied","Data":"d3b2d1de32476825bb4e0d834fe6bcf18ed45047a2f3de0c78d85719a38f451d"}
Nov 03 13:15:18.495897 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.495429    4909 scope.go:122] "RemoveContainer" containerID="f43ffd7a11876740a74bbddf93609ec76efb2ddc3cc45033b88bed8e153d038a"
Nov 03 13:15:18.497573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.497544    4909 generic.go:356] "Generic (PLEG): container finished" podID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerID="2725736b5b5d3e658689ce93088013ed5d9834d55187cecd36474d4216105650" exitCode=60
Nov 03 13:15:18.497736 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.497609    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" event={"ID":"2d7c238c-75f3-4f39-b899-a46a57487fcc","Type":"ContainerDied","Data":"2725736b5b5d3e658689ce93088013ed5d9834d55187cecd36474d4216105650"}
Nov 03 13:15:18.498031 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.498015    4909 scope.go:122] "RemoveContainer" containerID="2725736b5b5d3e658689ce93088013ed5d9834d55187cecd36474d4216105650"
Nov 03 13:15:18.498292 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:18.498270    4909 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26_odh-ai-gateway-infra(2d7c238c-75f3-4f39-b899-a46a57487fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc"
Nov 03 13:15:18.502120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.502097    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"
Nov 03 13:15:18.502237 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.502101    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8" event={"ID":"a3957cb6-a26a-45f6-8b93-5abc02b422ec","Type":"ContainerDied","Data":"51e002c6d80c01bd84b1eeb539aa2d2cd335d41540b80c286d40788c0c2e1c17"}
Nov 03 13:15:18.504721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.504179    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr" event={"ID":"ef3638d1-389f-41b8-8fa1-0d3053a39f63","Type":"ContainerStarted","Data":"0f7b72568caf1cfbfea0b472bffe922c17f9151ebbf75549a5c3970ac2a4c1c1"}
Nov 03 13:15:18.504721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.504238    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr" event={"ID":"ef3638d1-389f-41b8-8fa1-0d3053a39f63","Type":"ContainerStarted","Data":"51a19935915b36c8938fb556d7a177c8d92f416e2f26ac94a46050cb2cc40628"}
Nov 03 13:15:18.507198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.507178    4909 scope.go:122] "RemoveContainer" containerID="011cfd0a14ca540a57df57c497f0f28f3a36a5786c7c20753c15cfb560a5aff7"
Nov 03 13:15:18.522499 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.522478    4909 scope.go:122] "RemoveContainer" containerID="31c62ff73a01601952c76c7f25d97ce00acf60faaef77cd08c1dbf668f06fc85"
Nov 03 13:15:18.621810 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.621769    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"]
Nov 03 13:15:18.646020 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.645982    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:18.678933 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.678857    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"]
Nov 03 13:15:18.682838 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.682814    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895195-vgxqs"]
Nov 03 13:15:18.740288 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.740261    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d788795-759c-423b-b487-f8e5e44028f0-kube-api-access-p295n\" (UniqueName: \"kubernetes.io/projected/2d788795-759c-423b-b487-f8e5e44028f0-kube-api-access-p295n\") pod \"2d788795-759c-423b-b487-f8e5e44028f0\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") "
Nov 03 13:15:18.740403 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.740357    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d788795-759c-423b-b487-f8e5e44028f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d788795-759c-423b-b487-f8e5e44028f0-maas-api-service-ca\") pod \"2d788795-759c-423b-b487-f8e5e44028f0\" (UID: \"2d788795-759c-423b-b487-f8e5e44028f0\") "
Nov 03 13:15:18.741429 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.741402    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d788795-759c-423b-b487-f8e5e44028f0-maas-api-service-ca" pod "2d788795-759c-423b-b487-f8e5e44028f0" (UID: "2d788795-759c-423b-b487-f8e5e44028f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:18.742809 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.742784    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d788795-759c-423b-b487-f8e5e44028f0-kube-api-access-p295n" pod "2d788795-759c-423b-b487-f8e5e44028f0" (UID: "2d788795-759c-423b-b487-f8e5e44028f0"). InnerVolumeSpecName "kube-api-access-p295n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:18.784407 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.784380    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"]
Nov 03 13:15:18.791022 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.790998    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895195-xd4v8"]
Nov 03 13:15:18.841840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.841813    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p295n\" (UniqueName: \"kubernetes.io/projected/2d788795-759c-423b-b487-f8e5e44028f0-kube-api-access-p295n\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:18.841840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:18.841837    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d788795-759c-423b-b487-f8e5e44028f0-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:19.004009 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.003951    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr" podStartSLOduration=3.003936104 podStartE2EDuration="3.003936104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:19.001877128 +0000 UTC m=+5068.014343791" watchObservedRunningTime="2026-11-03 13:15:19.003936104 +0000 UTC m=+5068.016402771"
Nov 03 13:15:19.125407 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.125319    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" path="/var/lib/kubelet/pods/141a8a03-06a4-49f2-8137-d49930e9e0ba/volumes"
Nov 03 13:15:19.125887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.125864    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" path="/var/lib/kubelet/pods/536be7e2-613f-49d1-b30a-bb3cd6228494/volumes"
Nov 03 13:15:19.126343 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.126323    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" path="/var/lib/kubelet/pods/a3957cb6-a26a-45f6-8b93-5abc02b422ec/volumes"
Nov 03 13:15:19.511603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.511574    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"
Nov 03 13:15:19.511603 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.511585    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw" event={"ID":"2d788795-759c-423b-b487-f8e5e44028f0","Type":"ContainerDied","Data":"3388b9f94cc006b8bc1b861cb11604dc560dc81f253984154682b0e4da696aae"}
Nov 03 13:15:19.512154 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.511654    4909 scope.go:122] "RemoveContainer" containerID="38f396b37f7bc722e6718bbb280beaea8adfa168ddb36c65899dac071a542f41"
Nov 03 13:15:19.514910 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.514890    4909 generic.go:356] "Generic (PLEG): container finished" podID="ef3638d1-389f-41b8-8fa1-0d3053a39f63" containerID="0f7b72568caf1cfbfea0b472bffe922c17f9151ebbf75549a5c3970ac2a4c1c1" exitCode=0
Nov 03 13:15:19.515047 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.514973    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr" event={"ID":"ef3638d1-389f-41b8-8fa1-0d3053a39f63","Type":"ContainerDied","Data":"0f7b72568caf1cfbfea0b472bffe922c17f9151ebbf75549a5c3970ac2a4c1c1"}
Nov 03 13:15:19.668952 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.668924    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"]
Nov 03 13:15:19.673880 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.673842    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895195-qnncw"]
Nov 03 13:15:19.680153 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.680131    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:19.695884 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.695859    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" podUID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:19.853604 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.853536    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7c238c-75f3-4f39-b899-a46a57487fcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c238c-75f3-4f39-b899-a46a57487fcc-maas-api-service-ca\") pod \"2d7c238c-75f3-4f39-b899-a46a57487fcc\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") "
Nov 03 13:15:19.853604 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.853574    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7c238c-75f3-4f39-b899-a46a57487fcc-kube-api-access-sb928\" (UniqueName: \"kubernetes.io/projected/2d7c238c-75f3-4f39-b899-a46a57487fcc-kube-api-access-sb928\") pod \"2d7c238c-75f3-4f39-b899-a46a57487fcc\" (UID: \"2d7c238c-75f3-4f39-b899-a46a57487fcc\") "
Nov 03 13:15:19.854003 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.853972    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7c238c-75f3-4f39-b899-a46a57487fcc-maas-api-service-ca" pod "2d7c238c-75f3-4f39-b899-a46a57487fcc" (UID: "2d7c238c-75f3-4f39-b899-a46a57487fcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:19.855793 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.855770    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7c238c-75f3-4f39-b899-a46a57487fcc-kube-api-access-sb928" pod "2d7c238c-75f3-4f39-b899-a46a57487fcc" (UID: "2d7c238c-75f3-4f39-b899-a46a57487fcc"). InnerVolumeSpecName "kube-api-access-sb928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:19.954875 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.954851    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c238c-75f3-4f39-b899-a46a57487fcc-maas-api-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:19.954875 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:19.954876    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb928\" (UniqueName: \"kubernetes.io/projected/2d7c238c-75f3-4f39-b899-a46a57487fcc-kube-api-access-sb928\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.535440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.535407    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"
Nov 03 13:15:20.535440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.535420    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26" event={"ID":"2d7c238c-75f3-4f39-b899-a46a57487fcc","Type":"ContainerDied","Data":"1d133306de713b3794019c5d790d829ce65085da34754e29bd987560073fd980"}
Nov 03 13:15:20.535959 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.535505    4909 scope.go:122] "RemoveContainer" containerID="2725736b5b5d3e658689ce93088013ed5d9834d55187cecd36474d4216105650"
Nov 03 13:15:20.698740 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.698005    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"]
Nov 03 13:15:20.705794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.705767    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" podUID="ded63b27-8f76-41d1-a992-38982f6a69b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:20.710635 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.710606    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:20.714481 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.714455    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895195-vhs26"]
Nov 03 13:15:20.869145 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.869062    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3638d1-389f-41b8-8fa1-0d3053a39f63-kube-api-access-ccjg6\" (UniqueName: \"kubernetes.io/projected/ef3638d1-389f-41b8-8fa1-0d3053a39f63-kube-api-access-ccjg6\") pod \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") "
Nov 03 13:15:20.869304 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.869256    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-openshift-ca\") pod \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") "
Nov 03 13:15:20.869377 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.869307    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-xks-ca\") pod \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\" (UID: \"ef3638d1-389f-41b8-8fa1-0d3053a39f63\") "
Nov 03 13:15:20.869783 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.869756    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-openshift-ca" pod "ef3638d1-389f-41b8-8fa1-0d3053a39f63" (UID: "ef3638d1-389f-41b8-8fa1-0d3053a39f63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:20.870194 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.870174    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-openshift-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.872381 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.872357    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-xks-ca" pod "ef3638d1-389f-41b8-8fa1-0d3053a39f63" (UID: "ef3638d1-389f-41b8-8fa1-0d3053a39f63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:20.873100 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.873073    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3638d1-389f-41b8-8fa1-0d3053a39f63-kube-api-access-ccjg6" pod "ef3638d1-389f-41b8-8fa1-0d3053a39f63" (UID: "ef3638d1-389f-41b8-8fa1-0d3053a39f63"). InnerVolumeSpecName "kube-api-access-ccjg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:20.971756 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.971691    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccjg6\" (UniqueName: \"kubernetes.io/projected/ef3638d1-389f-41b8-8fa1-0d3053a39f63-kube-api-access-ccjg6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.971756 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:20.971753    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3638d1-389f-41b8-8fa1-0d3053a39f63-maas-api-xks-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.124471 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:21.124396    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d788795-759c-423b-b487-f8e5e44028f0" path="/var/lib/kubelet/pods/2d788795-759c-423b-b487-f8e5e44028f0/volumes"
Nov 03 13:15:21.124834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:21.124820    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" path="/var/lib/kubelet/pods/2d7c238c-75f3-4f39-b899-a46a57487fcc/volumes"
Nov 03 13:15:21.545510 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:21.545479    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"
Nov 03 13:15:21.545953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:21.545513    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr" event={"ID":"ef3638d1-389f-41b8-8fa1-0d3053a39f63","Type":"ContainerDied","Data":"51a19935915b36c8938fb556d7a177c8d92f416e2f26ac94a46050cb2cc40628"}
Nov 03 13:15:21.545953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:21.545564    4909 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a19935915b36c8938fb556d7a177c8d92f416e2f26ac94a46050cb2cc40628"
Nov 03 13:15:25.004914 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.004874    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" podUID="cc822e2b-def1-4eb9-a22b-f03dc395205c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:25.325173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.325146    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 13:15:25.426369 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.426342    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 13:15:25.426683 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.426663    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded63b27-8f76-41d1-a992-38982f6a69b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded63b27-8f76-41d1-a992-38982f6a69b4-plugins-config-volume\") pod \"ded63b27-8f76-41d1-a992-38982f6a69b4\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") "
Nov 03 13:15:25.426793 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.426778    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded63b27-8f76-41d1-a992-38982f6a69b4-kube-api-access-vw6ct\" (UniqueName: \"kubernetes.io/projected/ded63b27-8f76-41d1-a992-38982f6a69b4-kube-api-access-vw6ct\") pod \"ded63b27-8f76-41d1-a992-38982f6a69b4\" (UID: \"ded63b27-8f76-41d1-a992-38982f6a69b4\") "
Nov 03 13:15:25.427026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.427005    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded63b27-8f76-41d1-a992-38982f6a69b4-plugins-config-volume" pod "ded63b27-8f76-41d1-a992-38982f6a69b4" (UID: "ded63b27-8f76-41d1-a992-38982f6a69b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:25.427197 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.427177    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded63b27-8f76-41d1-a992-38982f6a69b4-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:25.428902 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.428885    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded63b27-8f76-41d1-a992-38982f6a69b4-kube-api-access-vw6ct" pod "ded63b27-8f76-41d1-a992-38982f6a69b4" (UID: "ded63b27-8f76-41d1-a992-38982f6a69b4"). InnerVolumeSpecName "kube-api-access-vw6ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:25.528251 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.528217    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0508f102-e252-43f4-a8a9-58fe6b0a06e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0508f102-e252-43f4-a8a9-58fe6b0a06e3-plugins-config-volume\") pod \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") "
Nov 03 13:15:25.528413 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.528381    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0508f102-e252-43f4-a8a9-58fe6b0a06e3-kube-api-access-6tl2v\" (UniqueName: \"kubernetes.io/projected/0508f102-e252-43f4-a8a9-58fe6b0a06e3-kube-api-access-6tl2v\") pod \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\" (UID: \"0508f102-e252-43f4-a8a9-58fe6b0a06e3\") "
Nov 03 13:15:25.528649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.528623    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0508f102-e252-43f4-a8a9-58fe6b0a06e3-plugins-config-volume" pod "0508f102-e252-43f4-a8a9-58fe6b0a06e3" (UID: "0508f102-e252-43f4-a8a9-58fe6b0a06e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:25.528746 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.528692    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6ct\" (UniqueName: \"kubernetes.io/projected/ded63b27-8f76-41d1-a992-38982f6a69b4-kube-api-access-vw6ct\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:25.530373 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.530352    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0508f102-e252-43f4-a8a9-58fe6b0a06e3-kube-api-access-6tl2v" pod "0508f102-e252-43f4-a8a9-58fe6b0a06e3" (UID: "0508f102-e252-43f4-a8a9-58fe6b0a06e3"). InnerVolumeSpecName "kube-api-access-6tl2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:25.573893 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.573821    4909 generic.go:356] "Generic (PLEG): container finished" podID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" containerID="8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962" exitCode=0
Nov 03 13:15:25.573893 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.573864    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"
Nov 03 13:15:25.574041 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.573900    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" event={"ID":"0508f102-e252-43f4-a8a9-58fe6b0a06e3","Type":"ContainerDied","Data":"8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962"}
Nov 03 13:15:25.574041 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.573954    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d" event={"ID":"0508f102-e252-43f4-a8a9-58fe6b0a06e3","Type":"ContainerDied","Data":"51586126d13a04f7e55bc96ab06d7f9f0b0a9786c8ea60731463cfc775dfb6fb"}
Nov 03 13:15:25.574041 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.573977    4909 scope.go:122] "RemoveContainer" containerID="8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962"
Nov 03 13:15:25.575647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.575625    4909 generic.go:356] "Generic (PLEG): container finished" podID="ded63b27-8f76-41d1-a992-38982f6a69b4" containerID="bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc" exitCode=0
Nov 03 13:15:25.575806 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.575675    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" event={"ID":"ded63b27-8f76-41d1-a992-38982f6a69b4","Type":"ContainerDied","Data":"bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc"}
Nov 03 13:15:25.575806 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.575681    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"
Nov 03 13:15:25.575806 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.575746    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk" event={"ID":"ded63b27-8f76-41d1-a992-38982f6a69b4","Type":"ContainerDied","Data":"afeb623d1fc9e4ad19f1caa53c083d155ecd9329a8c8f67ddccd890d25288bf1"}
Nov 03 13:15:25.586565 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.586538    4909 scope.go:122] "RemoveContainer" containerID="8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962"
Nov 03 13:15:25.586942 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:25.586918    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962\": container with ID starting with 8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962 not found: ID does not exist" containerID="8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962"
Nov 03 13:15:25.587038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.586953    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962"} err="failed to get container status \"8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962\": rpc error: code = NotFound desc = could not find container \"8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962\": container with ID starting with 8108ce247d735993f96f597b02b8fb6d7f4efc2794bcfec6f18ba09b9b284962 not found: ID does not exist"
Nov 03 13:15:25.587038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.586979    4909 scope.go:122] "RemoveContainer" containerID="bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc"
Nov 03 13:15:25.598463 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.598443    4909 scope.go:122] "RemoveContainer" containerID="bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc"
Nov 03 13:15:25.598792 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:25.598765    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc\": container with ID starting with bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc not found: ID does not exist" containerID="bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc"
Nov 03 13:15:25.598860 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.598806    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc"} err="failed to get container status \"bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc\": rpc error: code = NotFound desc = could not find container \"bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc\": container with ID starting with bce20d43055d2390ac6ba104e51c9a1fd816116c0694796567bd3f92f1b7f5cc not found: ID does not exist"
Nov 03 13:15:25.629746 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.629718    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tl2v\" (UniqueName: \"kubernetes.io/projected/0508f102-e252-43f4-a8a9-58fe6b0a06e3-kube-api-access-6tl2v\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:25.629853 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.629748    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0508f102-e252-43f4-a8a9-58fe6b0a06e3-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:25.726999 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.726965    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"]
Nov 03 13:15:25.736083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.736053    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1250b3-76ff5b9cbc-xxg6d"]
Nov 03 13:15:25.835775 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.835412    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"]
Nov 03 13:15:25.853373 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:25.853339    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1250b3-67fcc65b85-gqvvk"]
Nov 03 13:15:26.010212 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:26.010176    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" podUID="4958a745-d677-4897-8d31-cf29f2a67b9c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:26.260151 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:26.260120    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"]
Nov 03 13:15:26.260333 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:26.260309    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" containerID="cri-o://3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0" gracePeriod=30
Nov 03 13:15:27.123195 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:27.123165    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" path="/var/lib/kubelet/pods/0508f102-e252-43f4-a8a9-58fe6b0a06e3/volumes"
Nov 03 13:15:27.123599 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:27.123584    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded63b27-8f76-41d1-a992-38982f6a69b4" path="/var/lib/kubelet/pods/ded63b27-8f76-41d1-a992-38982f6a69b4/volumes"
Nov 03 13:15:27.303295 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:27.303262    4909 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:15:27.303457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:27.303322    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:15:31.527490 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.527423    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:15:31.606023 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.605994    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2059f309-d7d6-4d00-b3cb-4d554b564068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2059f309-d7d6-4d00-b3cb-4d554b564068-istiod-ca-cert\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606159 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606039    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-credential-socket\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606159 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606075    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-envoy\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606256 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606233    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-socket\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606299 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606277    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-certs\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606377 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606343    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-credential-socket" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.606452 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606363    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-token\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-token\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606452 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606381    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2059f309-d7d6-4d00-b3cb-4d554b564068-istiod-ca-cert" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:31.606592 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606458    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-kube-api-access-dhk6q\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-kube-api-access-dhk6q\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606592 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606469    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-socket" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.606592 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606563    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-podinfo\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606782 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606596    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-data\") pod \"2059f309-d7d6-4d00-b3cb-4d554b564068\" (UID: \"2059f309-d7d6-4d00-b3cb-4d554b564068\") "
Nov 03 13:15:31.606782 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606616    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-certs" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.607012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.606951    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-data" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.607200 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.607182    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.607270 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.607207    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.607270 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.607222    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.607270 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.607235    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.607270 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.607249    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2059f309-d7d6-4d00-b3cb-4d554b564068-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.608327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.608303    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-envoy" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.608626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.608603    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-podinfo" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:15:31.608857 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.608840    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-token" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:31.608955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.608867    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-kube-api-access-dhk6q" pod "2059f309-d7d6-4d00-b3cb-4d554b564068" (UID: "2059f309-d7d6-4d00-b3cb-4d554b564068"). InnerVolumeSpecName "kube-api-access-dhk6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:31.617459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.617434    4909 generic.go:356] "Generic (PLEG): container finished" podID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerID="3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0" exitCode=0
Nov 03 13:15:31.617576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.617477    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"
Nov 03 13:15:31.617576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.617512    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" event={"ID":"2059f309-d7d6-4d00-b3cb-4d554b564068","Type":"ContainerDied","Data":"3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0"}
Nov 03 13:15:31.617576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.617574    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6" event={"ID":"2059f309-d7d6-4d00-b3cb-4d554b564068","Type":"ContainerDied","Data":"3d4b28d2a8528d87faef185042206b3e892f9d3aafd587bc6e85e99a3100763c"}
Nov 03 13:15:31.617729 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.617603    4909 scope.go:122] "RemoveContainer" containerID="3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0"
Nov 03 13:15:31.639268 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.639248    4909 scope.go:122] "RemoveContainer" containerID="3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0"
Nov 03 13:15:31.639524 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:31.639506    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0\": container with ID starting with 3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0 not found: ID does not exist" containerID="3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0"
Nov 03 13:15:31.639571 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.639532    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0"} err="failed to get container status \"3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0\": rpc error: code = NotFound desc = could not find container \"3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0\": container with ID starting with 3d65ce0379c1f107bdcee1297d0088da78ba5a52ecb2983877c04e949b9f0ad0 not found: ID does not exist"
Nov 03 13:15:31.708274 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.708246    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.708274 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.708271    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.708439 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.708281    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.708439 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.708291    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhk6q\" (UniqueName: \"kubernetes.io/projected/2059f309-d7d6-4d00-b3cb-4d554b564068-kube-api-access-dhk6q\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.788064 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.787999    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"]
Nov 03 13:15:31.791449 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:31.791425    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-4g4p6"]
Nov 03 13:15:33.123786 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:33.123754    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" path="/var/lib/kubelet/pods/2059f309-d7d6-4d00-b3cb-4d554b564068/volumes"
Nov 03 13:15:35.004956 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:35.004921    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" podUID="cc822e2b-def1-4eb9-a22b-f03dc395205c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:36.021751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.020391    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" podUID="4958a745-d677-4897-8d31-cf29f2a67b9c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:36.401998 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.401967    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 13:15:36.462644 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.462619    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4958a745-d677-4897-8d31-cf29f2a67b9c-kube-api-access-5mv56\" (UniqueName: \"kubernetes.io/projected/4958a745-d677-4897-8d31-cf29f2a67b9c-kube-api-access-5mv56\") pod \"4958a745-d677-4897-8d31-cf29f2a67b9c\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") "
Nov 03 13:15:36.462826 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.462807    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4958a745-d677-4897-8d31-cf29f2a67b9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4958a745-d677-4897-8d31-cf29f2a67b9c-plugins-config-volume\") pod \"4958a745-d677-4897-8d31-cf29f2a67b9c\" (UID: \"4958a745-d677-4897-8d31-cf29f2a67b9c\") "
Nov 03 13:15:36.463178 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.463153    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4958a745-d677-4897-8d31-cf29f2a67b9c-plugins-config-volume" pod "4958a745-d677-4897-8d31-cf29f2a67b9c" (UID: "4958a745-d677-4897-8d31-cf29f2a67b9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:36.463305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.463266    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4958a745-d677-4897-8d31-cf29f2a67b9c-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:36.465942 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.465920    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4958a745-d677-4897-8d31-cf29f2a67b9c-kube-api-access-5mv56" pod "4958a745-d677-4897-8d31-cf29f2a67b9c" (UID: "4958a745-d677-4897-8d31-cf29f2a67b9c"). InnerVolumeSpecName "kube-api-access-5mv56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:36.520477 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.520453    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 13:15:36.564537 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.564474    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc822e2b-def1-4eb9-a22b-f03dc395205c-kube-api-access-4sdr7\" (UniqueName: \"kubernetes.io/projected/cc822e2b-def1-4eb9-a22b-f03dc395205c-kube-api-access-4sdr7\") pod \"cc822e2b-def1-4eb9-a22b-f03dc395205c\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") "
Nov 03 13:15:36.564659 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.564534    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc822e2b-def1-4eb9-a22b-f03dc395205c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc822e2b-def1-4eb9-a22b-f03dc395205c-plugins-config-volume\") pod \"cc822e2b-def1-4eb9-a22b-f03dc395205c\" (UID: \"cc822e2b-def1-4eb9-a22b-f03dc395205c\") "
Nov 03 13:15:36.564920 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.564897    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc822e2b-def1-4eb9-a22b-f03dc395205c-plugins-config-volume" pod "cc822e2b-def1-4eb9-a22b-f03dc395205c" (UID: "cc822e2b-def1-4eb9-a22b-f03dc395205c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:36.565006 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.564961    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mv56\" (UniqueName: \"kubernetes.io/projected/4958a745-d677-4897-8d31-cf29f2a67b9c-kube-api-access-5mv56\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:36.566535 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.566516    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc822e2b-def1-4eb9-a22b-f03dc395205c-kube-api-access-4sdr7" pod "cc822e2b-def1-4eb9-a22b-f03dc395205c" (UID: "cc822e2b-def1-4eb9-a22b-f03dc395205c"). InnerVolumeSpecName "kube-api-access-4sdr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:36.648864 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.648832    4909 generic.go:356] "Generic (PLEG): container finished" podID="4958a745-d677-4897-8d31-cf29f2a67b9c" containerID="39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080" exitCode=0
Nov 03 13:15:36.649050 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.648889    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"
Nov 03 13:15:36.649050 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.648972    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" event={"ID":"4958a745-d677-4897-8d31-cf29f2a67b9c","Type":"ContainerDied","Data":"39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080"}
Nov 03 13:15:36.649050 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.649010    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74" event={"ID":"4958a745-d677-4897-8d31-cf29f2a67b9c","Type":"ContainerDied","Data":"0efb6cca3bc289d05545f58baf7615648bf40d49655db2336febee2f1a8da5f2"}
Nov 03 13:15:36.649050 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.649029    4909 scope.go:122] "RemoveContainer" containerID="39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080"
Nov 03 13:15:36.657684 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.657658    4909 generic.go:356] "Generic (PLEG): container finished" podID="cc822e2b-def1-4eb9-a22b-f03dc395205c" containerID="e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2" exitCode=0
Nov 03 13:15:36.657860 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.657742    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"
Nov 03 13:15:36.657860 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.657748    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" event={"ID":"cc822e2b-def1-4eb9-a22b-f03dc395205c","Type":"ContainerDied","Data":"e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2"}
Nov 03 13:15:36.657860 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.657796    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7" event={"ID":"cc822e2b-def1-4eb9-a22b-f03dc395205c","Type":"ContainerDied","Data":"c3929039df1e7ba254a6adc8f2f99ce95718548cd1c4a20a6022aa4e9a5ee0c4"}
Nov 03 13:15:36.665816 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.665799    4909 scope.go:122] "RemoveContainer" containerID="39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080"
Nov 03 13:15:36.666152 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:36.666100    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080\": container with ID starting with 39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080 not found: ID does not exist" containerID="39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080"
Nov 03 13:15:36.666152 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.666123    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sdr7\" (UniqueName: \"kubernetes.io/projected/cc822e2b-def1-4eb9-a22b-f03dc395205c-kube-api-access-4sdr7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:36.666152 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.666143    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc822e2b-def1-4eb9-a22b-f03dc395205c-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:36.666152 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.666143    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080"} err="failed to get container status \"39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080\": rpc error: code = NotFound desc = could not find container \"39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080\": container with ID starting with 39ab455f7052b82ddd6d0bc1a575204e18b2587d107c8fbcec5fd3f2a5c22080 not found: ID does not exist"
Nov 03 13:15:36.666459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.666165    4909 scope.go:122] "RemoveContainer" containerID="e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2"
Nov 03 13:15:36.675486 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.675458    4909 scope.go:122] "RemoveContainer" containerID="e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2"
Nov 03 13:15:36.675772 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:36.675748    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2\": container with ID starting with e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2 not found: ID does not exist" containerID="e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2"
Nov 03 13:15:36.675849 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.675792    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2"} err="failed to get container status \"e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2\": rpc error: code = NotFound desc = could not find container \"e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2\": container with ID starting with e7d62dcd20f6a0a9276e17efc25a328730343d7e316d16d6424e7e8ad2b0ebd2 not found: ID does not exist"
Nov 03 13:15:36.803055 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.803031    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"]
Nov 03 13:15:36.805509 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.805488    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cacb84-c76668bd7-m87g7"]
Nov 03 13:15:36.925271 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.925172    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"]
Nov 03 13:15:36.927069 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:36.927030    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cacb84-5c9ddb9ddc-v4w74"]
Nov 03 13:15:37.123547 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:37.123518    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4958a745-d677-4897-8d31-cf29f2a67b9c" path="/var/lib/kubelet/pods/4958a745-d677-4897-8d31-cf29f2a67b9c/volumes"
Nov 03 13:15:37.123975 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:37.123959    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc822e2b-def1-4eb9-a22b-f03dc395205c" path="/var/lib/kubelet/pods/cc822e2b-def1-4eb9-a22b-f03dc395205c/volumes"
Nov 03 13:15:39.564960 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:15:39.564925    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:15:51.804295 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:51.804264    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:15:51.814963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:51.814941    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:15:55.169155 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:55.169124    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"]
Nov 03 13:15:55.215197 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:55.215168    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"]
Nov 03 13:15:59.299825 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:15:59.299795    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"]
Nov 03 13:16:02.893813 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:02.893781    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"]
Nov 03 13:16:03.317676 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:03.317638    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:16:09.563663 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:16:09.563621    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:16:09.660343 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:09.660309    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"]
Nov 03 13:16:09.660515 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:09.660498    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl" podUID="8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" containerName="maas-api" containerID="cri-o://1a192d74ee32e7959dd9541d1c3fd02f042d36ea673526b0f50726e884203bbe" gracePeriod=30
Nov 03 13:16:09.897466 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:09.897410    4909 generic.go:356] "Generic (PLEG): container finished" podID="8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" containerID="1a192d74ee32e7959dd9541d1c3fd02f042d36ea673526b0f50726e884203bbe" exitCode=0
Nov 03 13:16:09.897768 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:09.897483    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl" event={"ID":"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5","Type":"ContainerDied","Data":"1a192d74ee32e7959dd9541d1c3fd02f042d36ea673526b0f50726e884203bbe"}
Nov 03 13:16:09.949653 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:09.949631    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 13:16:10.096120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.096032    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-kube-api-access-7nvgb\" (UniqueName: \"kubernetes.io/projected/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-kube-api-access-7nvgb\") pod \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") "
Nov 03 13:16:10.096294 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.096133    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-maas-api-tls\") pod \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") "
Nov 03 13:16:10.096294 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.096184    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-metrics-certs\") pod \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") "
Nov 03 13:16:10.096294 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.096274    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-openshift-service-ca\") pod \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\" (UID: \"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5\") "
Nov 03 13:16:10.096797 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.096741    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-openshift-service-ca" pod "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" (UID: "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:10.098316 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.098285    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-kube-api-access-7nvgb" pod "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" (UID: "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5"). InnerVolumeSpecName "kube-api-access-7nvgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:10.098737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.098687    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-maas-api-tls" pod "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" (UID: "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:10.098816 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.098758    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-metrics-certs" pod "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" (UID: "8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:10.190152 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.190101    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main" containerID="cri-o://e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1" gracePeriod=45
Nov 03 13:16:10.198500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.198466    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nvgb\" (UniqueName: \"kubernetes.io/projected/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-kube-api-access-7nvgb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.198629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.198506    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.198629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.198520    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.198629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.198534    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.235098 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.235049    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main" containerID="cri-o://7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822" gracePeriod=45
Nov 03 13:16:10.482563 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.482526    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 13:16:10.523597 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.523574    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 13:16:10.603342 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603268    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-dshm\") pod \"d8e314af-4ea2-419e-879c-eaa78cedb602\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") "
Nov 03 13:16:10.603753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603360    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-dshm\") pod \"f731ab06-b572-43e3-9fef-972fed967e57\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") "
Nov 03 13:16:10.603753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603393    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e314af-4ea2-419e-879c-eaa78cedb602-kube-api-access-8ds9l\" (UniqueName: \"kubernetes.io/projected/d8e314af-4ea2-419e-879c-eaa78cedb602-kube-api-access-8ds9l\") pod \"d8e314af-4ea2-419e-879c-eaa78cedb602\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") "
Nov 03 13:16:10.603753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603419    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-model-cache\") pod \"f731ab06-b572-43e3-9fef-972fed967e57\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") "
Nov 03 13:16:10.603753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603451    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f731ab06-b572-43e3-9fef-972fed967e57-tls-certs\" (UniqueName: \"kubernetes.io/secret/f731ab06-b572-43e3-9fef-972fed967e57-tls-certs\") pod \"f731ab06-b572-43e3-9fef-972fed967e57\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") "
Nov 03 13:16:10.603753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603721    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-model-cache\") pod \"d8e314af-4ea2-419e-879c-eaa78cedb602\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") "
Nov 03 13:16:10.603753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603742    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-model-cache" pod "f731ab06-b572-43e3-9fef-972fed967e57" (UID: "f731ab06-b572-43e3-9fef-972fed967e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.604136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603786    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-home\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-home\") pod \"f731ab06-b572-43e3-9fef-972fed967e57\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") "
Nov 03 13:16:10.604136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603934    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f731ab06-b572-43e3-9fef-972fed967e57-kube-api-access-mq6bl\" (UniqueName: \"kubernetes.io/projected/f731ab06-b572-43e3-9fef-972fed967e57-kube-api-access-mq6bl\") pod \"f731ab06-b572-43e3-9fef-972fed967e57\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") "
Nov 03 13:16:10.604136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603969    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-tmp-dir\") pod \"f731ab06-b572-43e3-9fef-972fed967e57\" (UID: \"f731ab06-b572-43e3-9fef-972fed967e57\") "
Nov 03 13:16:10.604136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.603976    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-model-cache" pod "d8e314af-4ea2-419e-879c-eaa78cedb602" (UID: "d8e314af-4ea2-419e-879c-eaa78cedb602"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.604136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604008    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e314af-4ea2-419e-879c-eaa78cedb602-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e314af-4ea2-419e-879c-eaa78cedb602-tls-certs\") pod \"d8e314af-4ea2-419e-879c-eaa78cedb602\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") "
Nov 03 13:16:10.604136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604080    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-home\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-home\") pod \"d8e314af-4ea2-419e-879c-eaa78cedb602\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") "
Nov 03 13:16:10.604433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604147    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-tmp-dir\") pod \"d8e314af-4ea2-419e-879c-eaa78cedb602\" (UID: \"d8e314af-4ea2-419e-879c-eaa78cedb602\") "
Nov 03 13:16:10.604665 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604645    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.604780 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604657    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-home" pod "f731ab06-b572-43e3-9fef-972fed967e57" (UID: "f731ab06-b572-43e3-9fef-972fed967e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.604780 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604670    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.604924 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604903    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-tmp-dir" pod "d8e314af-4ea2-419e-879c-eaa78cedb602" (UID: "d8e314af-4ea2-419e-879c-eaa78cedb602"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.604996 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.604928    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-tmp-dir" pod "f731ab06-b572-43e3-9fef-972fed967e57" (UID: "f731ab06-b572-43e3-9fef-972fed967e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.605182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.605159    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-home" pod "d8e314af-4ea2-419e-879c-eaa78cedb602" (UID: "d8e314af-4ea2-419e-879c-eaa78cedb602"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.606129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.606103    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-dshm" pod "f731ab06-b572-43e3-9fef-972fed967e57" (UID: "f731ab06-b572-43e3-9fef-972fed967e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.606129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.606120    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f731ab06-b572-43e3-9fef-972fed967e57-tls-certs" pod "f731ab06-b572-43e3-9fef-972fed967e57" (UID: "f731ab06-b572-43e3-9fef-972fed967e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:10.606278 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.606205    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-dshm" pod "d8e314af-4ea2-419e-879c-eaa78cedb602" (UID: "d8e314af-4ea2-419e-879c-eaa78cedb602"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.606545 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.606522    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e314af-4ea2-419e-879c-eaa78cedb602-kube-api-access-8ds9l" pod "d8e314af-4ea2-419e-879c-eaa78cedb602" (UID: "d8e314af-4ea2-419e-879c-eaa78cedb602"). InnerVolumeSpecName "kube-api-access-8ds9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:10.606955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.606930    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f731ab06-b572-43e3-9fef-972fed967e57-kube-api-access-mq6bl" pod "f731ab06-b572-43e3-9fef-972fed967e57" (UID: "f731ab06-b572-43e3-9fef-972fed967e57"). InnerVolumeSpecName "kube-api-access-mq6bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:10.607025 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.607007    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e314af-4ea2-419e-879c-eaa78cedb602-tls-certs" pod "d8e314af-4ea2-419e-879c-eaa78cedb602" (UID: "d8e314af-4ea2-419e-879c-eaa78cedb602"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:10.706612 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706569    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6bl\" (UniqueName: \"kubernetes.io/projected/f731ab06-b572-43e3-9fef-972fed967e57-kube-api-access-mq6bl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706612 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706609    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706624    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e314af-4ea2-419e-879c-eaa78cedb602-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706637    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706651    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706663    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e314af-4ea2-419e-879c-eaa78cedb602-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706675    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706688    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ds9l\" (UniqueName: \"kubernetes.io/projected/d8e314af-4ea2-419e-879c-eaa78cedb602-kube-api-access-8ds9l\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706734    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f731ab06-b572-43e3-9fef-972fed967e57-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.706842 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.706747    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f731ab06-b572-43e3-9fef-972fed967e57-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.906467 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.906385    4909 generic.go:356] "Generic (PLEG): container finished" podID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerID="7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822" exitCode=0
Nov 03 13:16:10.906629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.906487    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"
Nov 03 13:16:10.906629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.906486    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" event={"ID":"d8e314af-4ea2-419e-879c-eaa78cedb602","Type":"ContainerDied","Data":"7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822"}
Nov 03 13:16:10.906765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.906635    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5" event={"ID":"d8e314af-4ea2-419e-879c-eaa78cedb602","Type":"ContainerDied","Data":"84e1c4c17643545ff1c7df293bcdefca61aac3d715a5477d4927636e77e85cc6"}
Nov 03 13:16:10.906765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.906666    4909 scope.go:122] "RemoveContainer" containerID="7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822"
Nov 03 13:16:10.908118 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.907980    4909 generic.go:356] "Generic (PLEG): container finished" podID="f731ab06-b572-43e3-9fef-972fed967e57" containerID="e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1" exitCode=0
Nov 03 13:16:10.908118 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.908007    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" event={"ID":"f731ab06-b572-43e3-9fef-972fed967e57","Type":"ContainerDied","Data":"e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1"}
Nov 03 13:16:10.908118 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.908035    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"
Nov 03 13:16:10.908118 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.908044    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t" event={"ID":"f731ab06-b572-43e3-9fef-972fed967e57","Type":"ContainerDied","Data":"bfc67f3b7e49ed0981db4b2473083a5104a641074f796ce2d7881da073e0aa99"}
Nov 03 13:16:10.909674 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.909650    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl" event={"ID":"8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5","Type":"ContainerDied","Data":"7839587e43fb65f4ffb54af240fb42741a37136bd36dfdb2994d3090805d1c9c"}
Nov 03 13:16:10.909897 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.909691    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"
Nov 03 13:16:10.918818 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.918800    4909 scope.go:122] "RemoveContainer" containerID="7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822"
Nov 03 13:16:10.919090 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:16:10.919074    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822\": container with ID starting with 7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822 not found: ID does not exist" containerID="7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822"
Nov 03 13:16:10.919153 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.919098    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822"} err="failed to get container status \"7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822\": rpc error: code = NotFound desc = could not find container \"7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822\": container with ID starting with 7f26d227515b049e69498f262a873cfc9d16b1d7d51a7895a6eb9d90996bd822 not found: ID does not exist"
Nov 03 13:16:10.919153 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.919112    4909 scope.go:122] "RemoveContainer" containerID="e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1"
Nov 03 13:16:10.927142 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.927119    4909 scope.go:122] "RemoveContainer" containerID="e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1"
Nov 03 13:16:10.927402 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:16:10.927382    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1\": container with ID starting with e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1 not found: ID does not exist" containerID="e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1"
Nov 03 13:16:10.927451 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.927409    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1"} err="failed to get container status \"e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1\": rpc error: code = NotFound desc = could not find container \"e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1\": container with ID starting with e2b3c99df499715f6ed0ece92a031219b875894c6775bc3f1cec0f8a94bf3eb1 not found: ID does not exist"
Nov 03 13:16:10.927451 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:10.927424    4909 scope.go:122] "RemoveContainer" containerID="1a192d74ee32e7959dd9541d1c3fd02f042d36ea673526b0f50726e884203bbe"
Nov 03 13:16:11.060552 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:11.060516    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"]
Nov 03 13:16:11.064239 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:11.064212    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-2-w3-bb1b51-kserve-655cd496d-v2g5t"]
Nov 03 13:16:11.123478 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:11.123453    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f731ab06-b572-43e3-9fef-972fed967e57" path="/var/lib/kubelet/pods/f731ab06-b572-43e3-9fef-972fed967e57/volumes"
Nov 03 13:16:11.167839 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:11.167776    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"]
Nov 03 13:16:11.172037 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:11.172010    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cacb84-765b8884d5-sq8jl"]
Nov 03 13:16:11.273184 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:11.273159    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"]
Nov 03 13:16:11.278440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:11.278419    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/simulator-w3-bb1b51-kserve-5d5d68d4fc-7ssb5"]
Nov 03 13:16:13.124405 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:13.124363    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" path="/var/lib/kubelet/pods/8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5/volumes"
Nov 03 13:16:13.124815 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:13.124806    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" path="/var/lib/kubelet/pods/d8e314af-4ea2-419e-879c-eaa78cedb602/volumes"
Nov 03 13:16:14.319848 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.319807    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m" podUID="4b1569f9-71ca-42c0-b16d-4d59b9cd7764" containerName="main" containerID="cri-o://0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5" gracePeriod=45
Nov 03 13:16:14.590752 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.590693    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:16:14.752001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.751969    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-home\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-home\") pod \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") "
Nov 03 13:16:14.752182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752080    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tls-certs\") pod \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") "
Nov 03 13:16:14.752182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752099    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tmp-dir\") pod \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") "
Nov 03 13:16:14.752182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752161    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-dshm\") pod \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") "
Nov 03 13:16:14.752386 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752265    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-kube-api-access-824ts\") pod \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") "
Nov 03 13:16:14.752386 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752280    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-home" pod "4b1569f9-71ca-42c0-b16d-4d59b9cd7764" (UID: "4b1569f9-71ca-42c0-b16d-4d59b9cd7764"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:14.752386 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752294    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-model-cache\") pod \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\" (UID: \"4b1569f9-71ca-42c0-b16d-4d59b9cd7764\") "
Nov 03 13:16:14.752558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752463    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tmp-dir" pod "4b1569f9-71ca-42c0-b16d-4d59b9cd7764" (UID: "4b1569f9-71ca-42c0-b16d-4d59b9cd7764"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:14.752558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752501    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-model-cache" pod "4b1569f9-71ca-42c0-b16d-4d59b9cd7764" (UID: "4b1569f9-71ca-42c0-b16d-4d59b9cd7764"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:14.752819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752801    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:14.752819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752819    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:14.752991 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.752827    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:14.754467 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.754445    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-dshm" pod "4b1569f9-71ca-42c0-b16d-4d59b9cd7764" (UID: "4b1569f9-71ca-42c0-b16d-4d59b9cd7764"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:14.754557 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.754486    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tls-certs" pod "4b1569f9-71ca-42c0-b16d-4d59b9cd7764" (UID: "4b1569f9-71ca-42c0-b16d-4d59b9cd7764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:14.754675 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.754659    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-kube-api-access-824ts" pod "4b1569f9-71ca-42c0-b16d-4d59b9cd7764" (UID: "4b1569f9-71ca-42c0-b16d-4d59b9cd7764"). InnerVolumeSpecName "kube-api-access-824ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:14.853765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.853687    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-kube-api-access-824ts\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:14.853765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.853726    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:14.853765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.853735    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1569f9-71ca-42c0-b16d-4d59b9cd7764-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:14.937247 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.937215    4909 generic.go:356] "Generic (PLEG): container finished" podID="4b1569f9-71ca-42c0-b16d-4d59b9cd7764" containerID="0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5" exitCode=0
Nov 03 13:16:14.937420 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.937258    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"
Nov 03 13:16:14.937420 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.937300    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m" event={"ID":"4b1569f9-71ca-42c0-b16d-4d59b9cd7764","Type":"ContainerDied","Data":"0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5"}
Nov 03 13:16:14.937420 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.937344    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m" event={"ID":"4b1569f9-71ca-42c0-b16d-4d59b9cd7764","Type":"ContainerDied","Data":"ec18cb4dda599cd611b326fe614a5363e6afb26c4dd0fbb3db720745dd38be48"}
Nov 03 13:16:14.937420 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.937364    4909 scope.go:122] "RemoveContainer" containerID="0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5"
Nov 03 13:16:14.947208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.947192    4909 scope.go:122] "RemoveContainer" containerID="0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5"
Nov 03 13:16:14.947472 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:16:14.947451    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5\": container with ID starting with 0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5 not found: ID does not exist" containerID="0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5"
Nov 03 13:16:14.947536 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:14.947480    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5"} err="failed to get container status \"0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5\": rpc error: code = NotFound desc = could not find container \"0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5\": container with ID starting with 0ab9ff2363c1523fb2dcdd145859773527fef4e8fb4ebd804cf26823464c83b5 not found: ID does not exist"
Nov 03 13:16:15.089223 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:15.089190    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"]
Nov 03 13:16:15.092851 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:15.092823    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/sub-test-model-a4489c-kserve-7865fbb6cf-29n5m"]
Nov 03 13:16:15.123771 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:15.123694    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1569f9-71ca-42c0-b16d-4d59b9cd7764" path="/var/lib/kubelet/pods/4b1569f9-71ca-42c0-b16d-4d59b9cd7764/volumes"
Nov 03 13:16:17.914677 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:17.914632    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9" podUID="71fcf678-1da0-47bf-b12d-714d130fc6ea" containerName="main" containerID="cri-o://c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609" gracePeriod=45
Nov 03 13:16:18.182306 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.182250    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:16:18.287167 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287136    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-tmp-dir\") pod \"71fcf678-1da0-47bf-b12d-714d130fc6ea\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") "
Nov 03 13:16:18.287325 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287250    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fcf678-1da0-47bf-b12d-714d130fc6ea-kube-api-access-lxlsj\" (UniqueName: \"kubernetes.io/projected/71fcf678-1da0-47bf-b12d-714d130fc6ea-kube-api-access-lxlsj\") pod \"71fcf678-1da0-47bf-b12d-714d130fc6ea\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") "
Nov 03 13:16:18.287325 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287314    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71fcf678-1da0-47bf-b12d-714d130fc6ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/71fcf678-1da0-47bf-b12d-714d130fc6ea-tls-certs\") pod \"71fcf678-1da0-47bf-b12d-714d130fc6ea\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") "
Nov 03 13:16:18.287464 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287343    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-model-cache\") pod \"71fcf678-1da0-47bf-b12d-714d130fc6ea\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") "
Nov 03 13:16:18.287464 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287398    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-tmp-dir" pod "71fcf678-1da0-47bf-b12d-714d130fc6ea" (UID: "71fcf678-1da0-47bf-b12d-714d130fc6ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:18.287464 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287445    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-home\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-home\") pod \"71fcf678-1da0-47bf-b12d-714d130fc6ea\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") "
Nov 03 13:16:18.287635 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287480    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-dshm\") pod \"71fcf678-1da0-47bf-b12d-714d130fc6ea\" (UID: \"71fcf678-1da0-47bf-b12d-714d130fc6ea\") "
Nov 03 13:16:18.287635 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287580    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-model-cache" pod "71fcf678-1da0-47bf-b12d-714d130fc6ea" (UID: "71fcf678-1da0-47bf-b12d-714d130fc6ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:18.287800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287757    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-home" pod "71fcf678-1da0-47bf-b12d-714d130fc6ea" (UID: "71fcf678-1da0-47bf-b12d-714d130fc6ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:18.287969 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287950    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:18.288042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287969    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:18.288042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.287978    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:18.289877 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.289850    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71fcf678-1da0-47bf-b12d-714d130fc6ea-tls-certs" pod "71fcf678-1da0-47bf-b12d-714d130fc6ea" (UID: "71fcf678-1da0-47bf-b12d-714d130fc6ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:18.289986 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.289879    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fcf678-1da0-47bf-b12d-714d130fc6ea-kube-api-access-lxlsj" pod "71fcf678-1da0-47bf-b12d-714d130fc6ea" (UID: "71fcf678-1da0-47bf-b12d-714d130fc6ea"). InnerVolumeSpecName "kube-api-access-lxlsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:18.289986 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.289896    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-dshm" pod "71fcf678-1da0-47bf-b12d-714d130fc6ea" (UID: "71fcf678-1da0-47bf-b12d-714d130fc6ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:18.388577 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.388539    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlsj\" (UniqueName: \"kubernetes.io/projected/71fcf678-1da0-47bf-b12d-714d130fc6ea-kube-api-access-lxlsj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:18.388577 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.388571    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fcf678-1da0-47bf-b12d-714d130fc6ea-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:18.388829 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.388586    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71fcf678-1da0-47bf-b12d-714d130fc6ea-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:18.965911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.965870    4909 generic.go:356] "Generic (PLEG): container finished" podID="71fcf678-1da0-47bf-b12d-714d130fc6ea" containerID="c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609" exitCode=0
Nov 03 13:16:18.966459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.965944    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"
Nov 03 13:16:18.966459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.965958    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9" event={"ID":"71fcf678-1da0-47bf-b12d-714d130fc6ea","Type":"ContainerDied","Data":"c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609"}
Nov 03 13:16:18.966459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.966048    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9" event={"ID":"71fcf678-1da0-47bf-b12d-714d130fc6ea","Type":"ContainerDied","Data":"0fff81bfbe7eb96b6e63f586419e730b698e67af60d37581bf0b754f18e85b25"}
Nov 03 13:16:18.966459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.966073    4909 scope.go:122] "RemoveContainer" containerID="c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609"
Nov 03 13:16:18.978663 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.978644    4909 scope.go:122] "RemoveContainer" containerID="c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609"
Nov 03 13:16:18.978996 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:16:18.978975    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609\": container with ID starting with c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609 not found: ID does not exist" containerID="c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609"
Nov 03 13:16:18.979085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:18.979003    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609"} err="failed to get container status \"c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609\": rpc error: code = NotFound desc = could not find container \"c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609\": container with ID starting with c42bd85e46723d6d7f352c71157f6bec6ed8243434b7b347db4af931001e3609 not found: ID does not exist"
Nov 03 13:16:19.124719 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:19.124672    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"]
Nov 03 13:16:19.124889 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:19.124730    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6fb3a8/sub-test-model-6fb3a8-kserve-7cc76b7645-tkdn9"]
Nov 03 13:16:21.123527 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:21.123498    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fcf678-1da0-47bf-b12d-714d130fc6ea" path="/var/lib/kubelet/pods/71fcf678-1da0-47bf-b12d-714d130fc6ea/volumes"
Nov 03 13:16:39.564250 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:16:39.564204    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:16:48.093909 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.093876    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"]
Nov 03 13:16:48.094432 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.094080    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92" podUID="101f028b-4d7a-4293-a325-905a16670ba4" containerName="maas-api" containerID="cri-o://27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c" gracePeriod=30
Nov 03 13:16:48.391140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.391114    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 13:16:48.558022 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.557993    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls\") pod \"101f028b-4d7a-4293-a325-905a16670ba4\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") "
Nov 03 13:16:48.558201 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.558044    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101f028b-4d7a-4293-a325-905a16670ba4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101f028b-4d7a-4293-a325-905a16670ba4-openshift-service-ca\") pod \"101f028b-4d7a-4293-a325-905a16670ba4\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") "
Nov 03 13:16:48.558201 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.558161    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-metrics-certs\") pod \"101f028b-4d7a-4293-a325-905a16670ba4\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") "
Nov 03 13:16:48.558201 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.558190    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101f028b-4d7a-4293-a325-905a16670ba4-kube-api-access-mghqm\" (UniqueName: \"kubernetes.io/projected/101f028b-4d7a-4293-a325-905a16670ba4-kube-api-access-mghqm\") pod \"101f028b-4d7a-4293-a325-905a16670ba4\" (UID: \"101f028b-4d7a-4293-a325-905a16670ba4\") "
Nov 03 13:16:48.558483 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.558459    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101f028b-4d7a-4293-a325-905a16670ba4-openshift-service-ca" pod "101f028b-4d7a-4293-a325-905a16670ba4" (UID: "101f028b-4d7a-4293-a325-905a16670ba4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:48.560077 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.560050    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls" pod "101f028b-4d7a-4293-a325-905a16670ba4" (UID: "101f028b-4d7a-4293-a325-905a16670ba4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:48.560201 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.560152    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-metrics-certs" pod "101f028b-4d7a-4293-a325-905a16670ba4" (UID: "101f028b-4d7a-4293-a325-905a16670ba4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:48.560255 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.560207    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101f028b-4d7a-4293-a325-905a16670ba4-kube-api-access-mghqm" pod "101f028b-4d7a-4293-a325-905a16670ba4" (UID: "101f028b-4d7a-4293-a325-905a16670ba4"). InnerVolumeSpecName "kube-api-access-mghqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:48.659626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.659564    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:48.659626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.659588    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101f028b-4d7a-4293-a325-905a16670ba4-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:48.659626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.659598    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101f028b-4d7a-4293-a325-905a16670ba4-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:48.659626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:48.659606    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghqm\" (UniqueName: \"kubernetes.io/projected/101f028b-4d7a-4293-a325-905a16670ba4-kube-api-access-mghqm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:49.153719 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.153634    4909 generic.go:356] "Generic (PLEG): container finished" podID="101f028b-4d7a-4293-a325-905a16670ba4" containerID="27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c" exitCode=0
Nov 03 13:16:49.154241 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.153853    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92" event={"ID":"101f028b-4d7a-4293-a325-905a16670ba4","Type":"ContainerDied","Data":"27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c"}
Nov 03 13:16:49.154241 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.153901    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92" event={"ID":"101f028b-4d7a-4293-a325-905a16670ba4","Type":"ContainerDied","Data":"6d81d1c4b99622efc9614d100bc75cab605c72ffe62badf75be90bb37d91d540"}
Nov 03 13:16:49.154241 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.153923    4909 scope.go:122] "RemoveContainer" containerID="27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c"
Nov 03 13:16:49.154241 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.154050    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"
Nov 03 13:16:49.168810 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.168746    4909 scope.go:122] "RemoveContainer" containerID="27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c"
Nov 03 13:16:49.169055 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:16:49.169033    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c\": container with ID starting with 27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c not found: ID does not exist" containerID="27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c"
Nov 03 13:16:49.169132 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.169068    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c"} err="failed to get container status \"27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c\": rpc error: code = NotFound desc = could not find container \"27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c\": container with ID starting with 27be8ef6c9edd9de6a891b4d3a70a4ee9afd77688f490cfb1a67f6b32aec406c not found: ID does not exist"
Nov 03 13:16:49.308174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.308143    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"]
Nov 03 13:16:49.311491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:49.311469    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bb1b51-7d569475df-q6l92"]
Nov 03 13:16:51.019438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:51.019407    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:16:51.123829 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:51.123800    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101f028b-4d7a-4293-a325-905a16670ba4" path="/var/lib/kubelet/pods/101f028b-4d7a-4293-a325-905a16670ba4/volumes"
Nov 03 13:16:54.496822 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:54.496791    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"]
Nov 03 13:16:54.497241 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:54.496987    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" containerID="cri-o://42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604" gracePeriod=30
Nov 03 13:16:58.329934 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:58.329898    4909 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:16:58.330325 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:58.329956    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:16:59.784540 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.784511    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 13:16:59.873325 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873297    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-envoy\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873484 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873334    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-credential-socket\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873484 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873370    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-certs\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873484 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873390    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-data\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873664 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873486    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f62df49b-98a6-46dd-9f16-64466785b187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df49b-98a6-46dd-9f16-64466785b187-istio-podinfo\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873664 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873513    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62df49b-98a6-46dd-9f16-64466785b187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df49b-98a6-46dd-9f16-64466785b187-istiod-ca-cert\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873664 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873549    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-kube-api-access-b4klb\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-kube-api-access-b4klb\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873664 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873584    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-socket\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873664 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873615    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-istio-token\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-istio-token\") pod \"f62df49b-98a6-46dd-9f16-64466785b187\" (UID: \"f62df49b-98a6-46dd-9f16-64466785b187\") "
Nov 03 13:16:59.873664 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873637    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-credential-socket" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:59.874038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873676    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-certs" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:59.874038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873812    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-data" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:59.874038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873940    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-socket" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:59.874038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.873977    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62df49b-98a6-46dd-9f16-64466785b187-istiod-ca-cert" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:59.874262 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.874061    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df49b-98a6-46dd-9f16-64466785b187-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.874262 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.874079    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.874262 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.874095    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.874262 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.874113    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.874262 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.874128    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.875712 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.875676    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-envoy" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:59.875823 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.875686    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-kube-api-access-b4klb" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "kube-api-access-b4klb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:59.875823 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.875741    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f62df49b-98a6-46dd-9f16-64466785b187-istio-podinfo" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:16:59.875823 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.875803    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-istio-token" pod "f62df49b-98a6-46dd-9f16-64466785b187" (UID: "f62df49b-98a6-46dd-9f16-64466785b187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:59.975928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.975891    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df49b-98a6-46dd-9f16-64466785b187-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.975928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.975931    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df49b-98a6-46dd-9f16-64466785b187-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.976432 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.975949    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4klb\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-kube-api-access-b4klb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:59.976432 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:16:59.975968    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df49b-98a6-46dd-9f16-64466785b187-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.217684 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.217654    4909 generic.go:356] "Generic (PLEG): container finished" podID="f62df49b-98a6-46dd-9f16-64466785b187" containerID="42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604" exitCode=0
Nov 03 13:17:00.217893 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.217723    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"
Nov 03 13:17:00.217893 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.217750    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" event={"ID":"f62df49b-98a6-46dd-9f16-64466785b187","Type":"ContainerDied","Data":"42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604"}
Nov 03 13:17:00.217893 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.217812    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb" event={"ID":"f62df49b-98a6-46dd-9f16-64466785b187","Type":"ContainerDied","Data":"0170c7ee8ba86b4492894260f228bd97cf323c7f2a92eab0459533e848b8c42e"}
Nov 03 13:17:00.217893 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.217841    4909 scope.go:122] "RemoveContainer" containerID="42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604"
Nov 03 13:17:00.228009 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.227992    4909 scope.go:122] "RemoveContainer" containerID="42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604"
Nov 03 13:17:00.228262 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:00.228243    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604\": container with ID starting with 42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604 not found: ID does not exist" containerID="42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604"
Nov 03 13:17:00.228327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.228273    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604"} err="failed to get container status \"42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604\": rpc error: code = NotFound desc = could not find container \"42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604\": container with ID starting with 42ae4c1d56ebee8be6f9043c1496bf6c7d5d0aa881a7154158511ee5e19fd604 not found: ID does not exist"
Nov 03 13:17:00.381021 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.380991    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"]
Nov 03 13:17:00.426776 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:00.426747    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-fkbnb"]
Nov 03 13:17:01.122744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:01.122710    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62df49b-98a6-46dd-9f16-64466785b187" path="/var/lib/kubelet/pods/f62df49b-98a6-46dd-9f16-64466785b187/volumes"
Nov 03 13:17:03.758781 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:03.758750    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"]
Nov 03 13:17:03.759256 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:03.759015    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" podUID="53e5714c-34d7-4250-8aad-724b2915baf7" containerName="maas-api" containerID="cri-o://6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412" gracePeriod=30
Nov 03 13:17:04.024507 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.024449    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 13:17:04.130727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.125085    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e5714c-34d7-4250-8aad-724b2915baf7-kube-api-access-z9xlj\" (UniqueName: \"kubernetes.io/projected/53e5714c-34d7-4250-8aad-724b2915baf7-kube-api-access-z9xlj\") pod \"53e5714c-34d7-4250-8aad-724b2915baf7\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") "
Nov 03 13:17:04.130727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.125228    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e5714c-34d7-4250-8aad-724b2915baf7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e5714c-34d7-4250-8aad-724b2915baf7-openshift-service-ca\") pod \"53e5714c-34d7-4250-8aad-724b2915baf7\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") "
Nov 03 13:17:04.130727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.125274    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-metrics-certs\") pod \"53e5714c-34d7-4250-8aad-724b2915baf7\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") "
Nov 03 13:17:04.130727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.125367    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls\") pod \"53e5714c-34d7-4250-8aad-724b2915baf7\" (UID: \"53e5714c-34d7-4250-8aad-724b2915baf7\") "
Nov 03 13:17:04.130727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.129767    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e5714c-34d7-4250-8aad-724b2915baf7-openshift-service-ca" pod "53e5714c-34d7-4250-8aad-724b2915baf7" (UID: "53e5714c-34d7-4250-8aad-724b2915baf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:04.137739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.136728    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e5714c-34d7-4250-8aad-724b2915baf7-kube-api-access-z9xlj" pod "53e5714c-34d7-4250-8aad-724b2915baf7" (UID: "53e5714c-34d7-4250-8aad-724b2915baf7"). InnerVolumeSpecName "kube-api-access-z9xlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:04.137739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.136844    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-metrics-certs" pod "53e5714c-34d7-4250-8aad-724b2915baf7" (UID: "53e5714c-34d7-4250-8aad-724b2915baf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:04.138209 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.138177    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls" pod "53e5714c-34d7-4250-8aad-724b2915baf7" (UID: "53e5714c-34d7-4250-8aad-724b2915baf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:04.226856 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.226820    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9xlj\" (UniqueName: \"kubernetes.io/projected/53e5714c-34d7-4250-8aad-724b2915baf7-kube-api-access-z9xlj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:04.226856 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.226850    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e5714c-34d7-4250-8aad-724b2915baf7-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:04.226856 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.226861    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:04.227072 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.226871    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e5714c-34d7-4250-8aad-724b2915baf7-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:04.240606 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.240579    4909 generic.go:356] "Generic (PLEG): container finished" podID="53e5714c-34d7-4250-8aad-724b2915baf7" containerID="6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412" exitCode=0
Nov 03 13:17:04.240750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.240624    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"
Nov 03 13:17:04.240750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.240658    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" event={"ID":"53e5714c-34d7-4250-8aad-724b2915baf7","Type":"ContainerDied","Data":"6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412"}
Nov 03 13:17:04.240750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.240732    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" event={"ID":"53e5714c-34d7-4250-8aad-724b2915baf7","Type":"ContainerDied","Data":"30b94cde149d490707209a3b81064cfc7c1f8a027c7096d2bf1060de1aef3efe"}
Nov 03 13:17:04.240868 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.240757    4909 scope.go:122] "RemoveContainer" containerID="6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412"
Nov 03 13:17:04.251091 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.251071    4909 scope.go:122] "RemoveContainer" containerID="6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412"
Nov 03 13:17:04.251426 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:04.251401    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412\": container with ID starting with 6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412 not found: ID does not exist" containerID="6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412"
Nov 03 13:17:04.251493 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.251437    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412"} err="failed to get container status \"6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412\": rpc error: code = NotFound desc = could not find container \"6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412\": container with ID starting with 6de41150fb75a7283f3c5c45dca0e1ba6672ed23a7f78a0df4ba9ef912ec5412 not found: ID does not exist"
Nov 03 13:17:04.393539 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.393455    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"]
Nov 03 13:17:04.397593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:04.397566    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc"]
Nov 03 13:17:05.122682 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:05.122653    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e5714c-34d7-4250-8aad-724b2915baf7" path="/var/lib/kubelet/pods/53e5714c-34d7-4250-8aad-724b2915baf7/volumes"
Nov 03 13:17:06.967827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:06.967782    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e0615c-754df7969-8xkkc" podUID="53e5714c-34d7-4250-8aad-724b2915baf7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.148:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:17:09.571776 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:09.571740    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:17:09.597647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:09.597614    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"]
Nov 03 13:17:09.597885 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:09.597862    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy" containerID="cri-o://b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6" gracePeriod=30
Nov 03 13:17:14.883630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:14.883604    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 13:17:15.034678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.034639    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8698500-d104-43a2-bb62-aae06cc3a4df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8698500-d104-43a2-bb62-aae06cc3a4df-istiod-ca-cert\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.034882 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.034751    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-podinfo\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.034882 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.034782    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-envoy\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.034972 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.034928    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-certs\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.035030 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035013    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-credential-socket\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.035075 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035052    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-socket\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.035116 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035080    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8698500-d104-43a2-bb62-aae06cc3a4df-istiod-ca-cert" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:15.035116 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035094    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-token\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-token\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.035281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035134    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-data\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.035281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035166    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-kube-api-access-4fkwm\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-kube-api-access-4fkwm\") pod \"f8698500-d104-43a2-bb62-aae06cc3a4df\" (UID: \"f8698500-d104-43a2-bb62-aae06cc3a4df\") "
Nov 03 13:17:15.035281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035199    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-certs" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:15.035281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035258    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-credential-socket" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:15.035521 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035276    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-socket" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:15.035521 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035442    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-data" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:15.035660 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035644    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.035750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035666    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.035750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035682    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.035750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035712    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8698500-d104-43a2-bb62-aae06cc3a4df-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.035750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.035727    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.037135 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.037108    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-podinfo" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:17:15.037255 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.037144    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-envoy" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:15.037349 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.037328    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-kube-api-access-4fkwm" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "kube-api-access-4fkwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:15.037392 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.037359    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-token" pod "f8698500-d104-43a2-bb62-aae06cc3a4df" (UID: "f8698500-d104-43a2-bb62-aae06cc3a4df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:15.102900 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.102871    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"]
Nov 03 13:17:15.105194 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.105171    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"]
Nov 03 13:17:15.136903 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.136877    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.137018 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.136903    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fkwm\" (UniqueName: \"kubernetes.io/projected/f8698500-d104-43a2-bb62-aae06cc3a4df-kube-api-access-4fkwm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.137018 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.136917    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.137018 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.136930    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8698500-d104-43a2-bb62-aae06cc3a4df-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:15.301387 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.301360    4909 generic.go:356] "Generic (PLEG): container finished" podID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerID="b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6" exitCode=0
Nov 03 13:17:15.301554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.301405    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"
Nov 03 13:17:15.301554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.301442    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" event={"ID":"f8698500-d104-43a2-bb62-aae06cc3a4df","Type":"ContainerDied","Data":"b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6"}
Nov 03 13:17:15.301554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.301496    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5" event={"ID":"f8698500-d104-43a2-bb62-aae06cc3a4df","Type":"ContainerDied","Data":"5c36b83d5e2a4c787d69c9151c03d5eec61be03eb98a441f245d5dc8d767209f"}
Nov 03 13:17:15.301554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.301523    4909 scope.go:122] "RemoveContainer" containerID="b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6"
Nov 03 13:17:15.311003 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.310985    4909 scope.go:122] "RemoveContainer" containerID="b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd"
Nov 03 13:17:15.319341 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.319323    4909 scope.go:122] "RemoveContainer" containerID="b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6"
Nov 03 13:17:15.319950 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:15.319805    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6\": container with ID starting with b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6 not found: ID does not exist" containerID="b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6"
Nov 03 13:17:15.319950 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.319855    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6"} err="failed to get container status \"b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6\": rpc error: code = NotFound desc = could not find container \"b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6\": container with ID starting with b08b7cfc1c8278675ef49676ce3034ef32e1b64fbc71d9682f5940936bff85f6 not found: ID does not exist"
Nov 03 13:17:15.319950 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.319880    4909 scope.go:122] "RemoveContainer" containerID="b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd"
Nov 03 13:17:15.320209 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:15.320185    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd\": container with ID starting with b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd not found: ID does not exist" containerID="b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd"
Nov 03 13:17:15.320339 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.320217    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd"} err="failed to get container status \"b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd\": rpc error: code = NotFound desc = could not find container \"b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd\": container with ID starting with b2c546b01cb7617450160e49fc256a2bcfb66c54f009d766565b1d805d0b09bd not found: ID does not exist"
Nov 03 13:17:15.425352 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.425259    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"]
Nov 03 13:17:15.429475 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:15.429444    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e0615c-openshift-default-7576dd8765-2j9m5"]
Nov 03 13:17:17.123447 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:17.123414    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" path="/var/lib/kubelet/pods/f8698500-d104-43a2-bb62-aae06cc3a4df/volumes"
Nov 03 13:17:17.795334 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:17.795294    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:17:30.130546 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.125491    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" podUID="955970d0-cd2b-46b5-8d24-4cf06cf33989" containerName="main" containerID="cri-o://7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea" gracePeriod=45
Nov 03 13:17:30.131582 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.131336    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" containerID="cri-o://a84a7dccfbde364b8ae65ce90e7d86f728970a693826fa0162b35aefd16fe05d" gracePeriod=45
Nov 03 13:17:30.395435 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.395362    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 13:17:30.398213 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.398183    4909 generic.go:356] "Generic (PLEG): container finished" podID="589f3710-4603-4806-916d-d3e41a58c0f6" containerID="a84a7dccfbde364b8ae65ce90e7d86f728970a693826fa0162b35aefd16fe05d" exitCode=0
Nov 03 13:17:30.398332 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.398259    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" event={"ID":"589f3710-4603-4806-916d-d3e41a58c0f6","Type":"ContainerDied","Data":"a84a7dccfbde364b8ae65ce90e7d86f728970a693826fa0162b35aefd16fe05d"}
Nov 03 13:17:30.399501 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.399482    4909 generic.go:356] "Generic (PLEG): container finished" podID="955970d0-cd2b-46b5-8d24-4cf06cf33989" containerID="7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea" exitCode=0
Nov 03 13:17:30.399608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.399520    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"
Nov 03 13:17:30.399608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.399557    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" event={"ID":"955970d0-cd2b-46b5-8d24-4cf06cf33989","Type":"ContainerDied","Data":"7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea"}
Nov 03 13:17:30.399608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.399600    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj" event={"ID":"955970d0-cd2b-46b5-8d24-4cf06cf33989","Type":"ContainerDied","Data":"d16ebebc38e5f6f3674f9ccd46f3dc1fd34e04aa978b9a3a1db64cd1e01d1619"}
Nov 03 13:17:30.399780 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.399622    4909 scope.go:122] "RemoveContainer" containerID="7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea"
Nov 03 13:17:30.410818 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.410800    4909 scope.go:122] "RemoveContainer" containerID="7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea"
Nov 03 13:17:30.411117 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:30.411094    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea\": container with ID starting with 7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea not found: ID does not exist" containerID="7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea"
Nov 03 13:17:30.411192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.411132    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea"} err="failed to get container status \"7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea\": rpc error: code = NotFound desc = could not find container \"7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea\": container with ID starting with 7357d1be7c1cc082b4e42c2d951701620bc22332b87c3c6cd57a7f4f0b3c8dea not found: ID does not exist"
Nov 03 13:17:30.423233 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.423213    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 13:17:30.498517 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498494    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955970d0-cd2b-46b5-8d24-4cf06cf33989-tls-certs\" (UniqueName: \"kubernetes.io/secret/955970d0-cd2b-46b5-8d24-4cf06cf33989-tls-certs\") pod \"955970d0-cd2b-46b5-8d24-4cf06cf33989\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") "
Nov 03 13:17:30.498621 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498529    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-dshm\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-dshm\") pod \"955970d0-cd2b-46b5-8d24-4cf06cf33989\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") "
Nov 03 13:17:30.498621 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498581    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-model-cache\") pod \"955970d0-cd2b-46b5-8d24-4cf06cf33989\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") "
Nov 03 13:17:30.498621 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498618    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-tmp-dir\") pod \"955970d0-cd2b-46b5-8d24-4cf06cf33989\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") "
Nov 03 13:17:30.498769 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498664    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-home\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-home\") pod \"955970d0-cd2b-46b5-8d24-4cf06cf33989\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") "
Nov 03 13:17:30.498809 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498792    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955970d0-cd2b-46b5-8d24-4cf06cf33989-kube-api-access-rf9p8\" (UniqueName: \"kubernetes.io/projected/955970d0-cd2b-46b5-8d24-4cf06cf33989-kube-api-access-rf9p8\") pod \"955970d0-cd2b-46b5-8d24-4cf06cf33989\" (UID: \"955970d0-cd2b-46b5-8d24-4cf06cf33989\") "
Nov 03 13:17:30.498938 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498907    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-model-cache" pod "955970d0-cd2b-46b5-8d24-4cf06cf33989" (UID: "955970d0-cd2b-46b5-8d24-4cf06cf33989"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.499029 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.498940    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-tmp-dir" pod "955970d0-cd2b-46b5-8d24-4cf06cf33989" (UID: "955970d0-cd2b-46b5-8d24-4cf06cf33989"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.499102 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.499032    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-home" pod "955970d0-cd2b-46b5-8d24-4cf06cf33989" (UID: "955970d0-cd2b-46b5-8d24-4cf06cf33989"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.499229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.499210    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.499307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.499235    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.499307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.499249    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.500933 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.500905    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-dshm" pod "955970d0-cd2b-46b5-8d24-4cf06cf33989" (UID: "955970d0-cd2b-46b5-8d24-4cf06cf33989"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.501036 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.500946    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955970d0-cd2b-46b5-8d24-4cf06cf33989-kube-api-access-rf9p8" pod "955970d0-cd2b-46b5-8d24-4cf06cf33989" (UID: "955970d0-cd2b-46b5-8d24-4cf06cf33989"). InnerVolumeSpecName "kube-api-access-rf9p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:30.501036 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.500956    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955970d0-cd2b-46b5-8d24-4cf06cf33989-tls-certs" pod "955970d0-cd2b-46b5-8d24-4cf06cf33989" (UID: "955970d0-cd2b-46b5-8d24-4cf06cf33989"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:30.599826 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.599797    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589f3710-4603-4806-916d-d3e41a58c0f6-kube-api-access-rv4hn\" (UniqueName: \"kubernetes.io/projected/589f3710-4603-4806-916d-d3e41a58c0f6-kube-api-access-rv4hn\") pod \"589f3710-4603-4806-916d-d3e41a58c0f6\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") "
Nov 03 13:17:30.600010 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.599844    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-tmp-dir\") pod \"589f3710-4603-4806-916d-d3e41a58c0f6\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") "
Nov 03 13:17:30.600010 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.599873    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-home\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-home\") pod \"589f3710-4603-4806-916d-d3e41a58c0f6\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") "
Nov 03 13:17:30.600010 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.599922    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-model-cache\") pod \"589f3710-4603-4806-916d-d3e41a58c0f6\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") "
Nov 03 13:17:30.600208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600016    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-dshm\") pod \"589f3710-4603-4806-916d-d3e41a58c0f6\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") "
Nov 03 13:17:30.600208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600083    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589f3710-4603-4806-916d-d3e41a58c0f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/589f3710-4603-4806-916d-d3e41a58c0f6-tls-certs\") pod \"589f3710-4603-4806-916d-d3e41a58c0f6\" (UID: \"589f3710-4603-4806-916d-d3e41a58c0f6\") "
Nov 03 13:17:30.600208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600163    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-home" pod "589f3710-4603-4806-916d-d3e41a58c0f6" (UID: "589f3710-4603-4806-916d-d3e41a58c0f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.600208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600182    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-tmp-dir" pod "589f3710-4603-4806-916d-d3e41a58c0f6" (UID: "589f3710-4603-4806-916d-d3e41a58c0f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.600208 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600195    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-model-cache" pod "589f3710-4603-4806-916d-d3e41a58c0f6" (UID: "589f3710-4603-4806-916d-d3e41a58c0f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.600491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600434    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9p8\" (UniqueName: \"kubernetes.io/projected/955970d0-cd2b-46b5-8d24-4cf06cf33989-kube-api-access-rf9p8\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.600491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600455    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.600491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600469    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.600491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600483    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955970d0-cd2b-46b5-8d24-4cf06cf33989-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.600732 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600495    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955970d0-cd2b-46b5-8d24-4cf06cf33989-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.600732 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.600509    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.602137 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.602108    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-dshm" pod "589f3710-4603-4806-916d-d3e41a58c0f6" (UID: "589f3710-4603-4806-916d-d3e41a58c0f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.602137 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.602125    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589f3710-4603-4806-916d-d3e41a58c0f6-tls-certs" pod "589f3710-4603-4806-916d-d3e41a58c0f6" (UID: "589f3710-4603-4806-916d-d3e41a58c0f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:30.602271 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.602224    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589f3710-4603-4806-916d-d3e41a58c0f6-kube-api-access-rv4hn" pod "589f3710-4603-4806-916d-d3e41a58c0f6" (UID: "589f3710-4603-4806-916d-d3e41a58c0f6"). InnerVolumeSpecName "kube-api-access-rv4hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:30.701219 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.701187    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/589f3710-4603-4806-916d-d3e41a58c0f6-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.701219 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.701216    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589f3710-4603-4806-916d-d3e41a58c0f6-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.701219 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.701226    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv4hn\" (UniqueName: \"kubernetes.io/projected/589f3710-4603-4806-916d-d3e41a58c0f6-kube-api-access-rv4hn\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.811972 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.811941    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"]
Nov 03 13:17:30.816792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:30.816766    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-2-w2-91cd99-kserve-5cb6c97647-fczbj"]
Nov 03 13:17:31.124133 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:31.124054    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955970d0-cd2b-46b5-8d24-4cf06cf33989" path="/var/lib/kubelet/pods/955970d0-cd2b-46b5-8d24-4cf06cf33989/volumes"
Nov 03 13:17:31.406229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:31.406144    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"
Nov 03 13:17:31.406229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:31.406174    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" event={"ID":"589f3710-4603-4806-916d-d3e41a58c0f6","Type":"ContainerDied","Data":"7470da4113aab5c3677472101790242521dbe91ea39dbc3bac151bd4997853dc"}
Nov 03 13:17:31.406229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:31.406223    4909 scope.go:122] "RemoveContainer" containerID="a84a7dccfbde364b8ae65ce90e7d86f728970a693826fa0162b35aefd16fe05d"
Nov 03 13:17:31.409108 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:31.409077    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 13:17:31.557903 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:31.557875    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"]
Nov 03 13:17:31.563769 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:31.563746    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/premium-w2-91cd99-kserve-85885d549c-nbvfc"]
Nov 03 13:17:33.122054 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:33.122022    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" path="/var/lib/kubelet/pods/589f3710-4603-4806-916d-d3e41a58c0f6/volumes"
Nov 03 13:17:35.211614 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:35.211583    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"]
Nov 03 13:17:35.212044 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:35.211793    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy" containerID="cri-o://5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e" gracePeriod=30
Nov 03 13:17:39.563221 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:39.563179    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:17:40.465545 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.465522    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 13:17:40.469853 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.469829    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"]
Nov 03 13:17:40.470014 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.469994    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" podUID="d11e72b2-5094-45a0-b147-7d86098e89c3" containerName="maas-api" containerID="cri-o://fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec" gracePeriod=30
Nov 03 13:17:40.479305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.479280    4909 generic.go:356] "Generic (PLEG): container finished" podID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerID="5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e" exitCode=0
Nov 03 13:17:40.479410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.479322    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"
Nov 03 13:17:40.479410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.479361    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" event={"ID":"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629","Type":"ContainerDied","Data":"5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e"}
Nov 03 13:17:40.479491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.479426    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56" event={"ID":"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629","Type":"ContainerDied","Data":"08a81c7b5ccfd0619649222f517c206f8fb3eccac1bbdd2adb0378a155c8165b"}
Nov 03 13:17:40.479491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.479455    4909 scope.go:122] "RemoveContainer" containerID="5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e"
Nov 03 13:17:40.512878 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.512849    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-socket\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.512974 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.512940    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-podinfo\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513020 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.512996    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-token\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-token\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513058 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513026    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-certs\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513057    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istiod-ca-cert\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513088    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-kube-api-access-j9skb\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-kube-api-access-j9skb\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513223 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513114    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-socket" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:40.513223 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513185    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-envoy\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513256    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-credential-socket\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513285    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-data\") pod \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\" (UID: \"7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629\") "
Nov 03 13:17:40.513558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513339    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-certs" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:40.513558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513433    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istiod-ca-cert" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:40.513876 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513566    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-credential-socket" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:40.513876 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513653    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-data" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:40.514338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513892    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.514338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513918    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.514338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513933    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.514338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513947    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.514338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.513959    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.516055 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.516029    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-token" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:40.516591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.516328    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-envoy" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:40.516591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.516352    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-kube-api-access-j9skb" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "kube-api-access-j9skb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:40.516591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.516402    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-podinfo" pod "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" (UID: "7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:17:40.584327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.584295    4909 scope.go:122] "RemoveContainer" containerID="2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c"
Nov 03 13:17:40.595468 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.595439    4909 scope.go:122] "RemoveContainer" containerID="5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e"
Nov 03 13:17:40.595840 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:40.595808    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e\": container with ID starting with 5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e not found: ID does not exist" containerID="5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e"
Nov 03 13:17:40.595953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.595840    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e"} err="failed to get container status \"5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e\": rpc error: code = NotFound desc = could not find container \"5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e\": container with ID starting with 5fac6748b31cdd7d78fe5c76f3adce9114024b56d923368a742d827cd1490e0e not found: ID does not exist"
Nov 03 13:17:40.595953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.595859    4909 scope.go:122] "RemoveContainer" containerID="2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c"
Nov 03 13:17:40.596146 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:40.596119    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c\": container with ID starting with 2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c not found: ID does not exist" containerID="2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c"
Nov 03 13:17:40.596217 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.596156    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c"} err="failed to get container status \"2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c\": rpc error: code = NotFound desc = could not find container \"2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c\": container with ID starting with 2d10802ce599cf8f4984268822a69bf113a3c3bb2bba04f4ab6d370c8014c98c not found: ID does not exist"
Nov 03 13:17:40.611827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.611787    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" podUID="d11e72b2-5094-45a0-b147-7d86098e89c3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.48:8443/health\": dial tcp 10.132.0.48:8443: connect: connection refused"
Nov 03 13:17:40.615478 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.615452    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.615478 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.615478    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.615666 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.615493    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9skb\" (UniqueName: \"kubernetes.io/projected/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-kube-api-access-j9skb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.615666 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.615505    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.719299 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.719273    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 13:17:40.816415 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.816387    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-metrics-certs\") pod \"d11e72b2-5094-45a0-b147-7d86098e89c3\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") "
Nov 03 13:17:40.816588 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.816429    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11e72b2-5094-45a0-b147-7d86098e89c3-kube-api-access-qzjkz\" (UniqueName: \"kubernetes.io/projected/d11e72b2-5094-45a0-b147-7d86098e89c3-kube-api-access-qzjkz\") pod \"d11e72b2-5094-45a0-b147-7d86098e89c3\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") "
Nov 03 13:17:40.816588 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.816463    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls\") pod \"d11e72b2-5094-45a0-b147-7d86098e89c3\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") "
Nov 03 13:17:40.816731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.816626    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11e72b2-5094-45a0-b147-7d86098e89c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11e72b2-5094-45a0-b147-7d86098e89c3-openshift-service-ca\") pod \"d11e72b2-5094-45a0-b147-7d86098e89c3\" (UID: \"d11e72b2-5094-45a0-b147-7d86098e89c3\") "
Nov 03 13:17:40.817034 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.817005    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11e72b2-5094-45a0-b147-7d86098e89c3-openshift-service-ca" pod "d11e72b2-5094-45a0-b147-7d86098e89c3" (UID: "d11e72b2-5094-45a0-b147-7d86098e89c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:40.818615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.818587    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-metrics-certs" pod "d11e72b2-5094-45a0-b147-7d86098e89c3" (UID: "d11e72b2-5094-45a0-b147-7d86098e89c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:40.818754 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.818628    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11e72b2-5094-45a0-b147-7d86098e89c3-kube-api-access-qzjkz" pod "d11e72b2-5094-45a0-b147-7d86098e89c3" (UID: "d11e72b2-5094-45a0-b147-7d86098e89c3"). InnerVolumeSpecName "kube-api-access-qzjkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:40.818754 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.818653    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls" pod "d11e72b2-5094-45a0-b147-7d86098e89c3" (UID: "d11e72b2-5094-45a0-b147-7d86098e89c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:40.917958 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.917932    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11e72b2-5094-45a0-b147-7d86098e89c3-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.917958 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.917959    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.918142 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.917970    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjkz\" (UniqueName: \"kubernetes.io/projected/d11e72b2-5094-45a0-b147-7d86098e89c3-kube-api-access-qzjkz\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.918142 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.917978    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11e72b2-5094-45a0-b147-7d86098e89c3-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:40.941181 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.941151    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"]
Nov 03 13:17:40.944347 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:40.944323    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a4489c-openshift-default-54f68c6dc-wjt56"]
Nov 03 13:17:41.122363 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.122291    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" path="/var/lib/kubelet/pods/7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629/volumes"
Nov 03 13:17:41.486075 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.486045    4909 generic.go:356] "Generic (PLEG): container finished" podID="d11e72b2-5094-45a0-b147-7d86098e89c3" containerID="fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec" exitCode=0
Nov 03 13:17:41.486244 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.486092    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"
Nov 03 13:17:41.486244 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.486133    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" event={"ID":"d11e72b2-5094-45a0-b147-7d86098e89c3","Type":"ContainerDied","Data":"fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec"}
Nov 03 13:17:41.486244 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.486192    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds" event={"ID":"d11e72b2-5094-45a0-b147-7d86098e89c3","Type":"ContainerDied","Data":"da9b659043148edbd5b8943dbee019026da9c06b36a688c52bfab0f788372ff9"}
Nov 03 13:17:41.486244 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.486218    4909 scope.go:122] "RemoveContainer" containerID="fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec"
Nov 03 13:17:41.495669 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.495650    4909 scope.go:122] "RemoveContainer" containerID="fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec"
Nov 03 13:17:41.495930 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:17:41.495910    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec\": container with ID starting with fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec not found: ID does not exist" containerID="fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec"
Nov 03 13:17:41.496000 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.495937    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec"} err="failed to get container status \"fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec\": rpc error: code = NotFound desc = could not find container \"fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec\": container with ID starting with fdceed886b5352c2f74f0e200523155166cf0f392971f2eb6dc6d85429c344ec not found: ID does not exist"
Nov 03 13:17:41.643135 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.643107    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"]
Nov 03 13:17:41.689292 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:41.689267    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-91cd99-675976cd6b-m75ds"]
Nov 03 13:17:42.462667 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:42.462635    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"]
Nov 03 13:17:42.574254 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:42.574221    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"]
Nov 03 13:17:43.122242 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:43.122204    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11e72b2-5094-45a0-b147-7d86098e89c3" path="/var/lib/kubelet/pods/d11e72b2-5094-45a0-b147-7d86098e89c3/volumes"
Nov 03 13:17:47.463592 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:47.463555    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" podUID="c931adf2-8c4f-4539-be64-6d5f17e08d96" containerName="payload-processing" containerID="cri-o://90ad301661cdc992b54a633ca207d902b76f06543fad4e2796e1f209482e8b64" gracePeriod=25
Nov 03 13:17:47.574726 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:47.574671    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" podUID="1e837472-8f8f-4682-ad78-0f7779e164f7" containerName="payload-pre-processing" containerID="cri-o://cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64" gracePeriod=25
Nov 03 13:17:53.179482 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:53.179442    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" podUID="1e837472-8f8f-4682-ad78-0f7779e164f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:54.185681 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:54.185640    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" podUID="c931adf2-8c4f-4539-be64-6d5f17e08d96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:59.045295 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:17:59.045261    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:18:03.178934 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:03.178896    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" podUID="1e837472-8f8f-4682-ad78-0f7779e164f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:04.185658 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:04.185624    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" podUID="c931adf2-8c4f-4539-be64-6d5f17e08d96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:07.297341 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.297306    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"]
Nov 03 13:18:07.297767 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.297519    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" containerID="cri-o://8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661" gracePeriod=30
Nov 03 13:18:07.629891 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.629862    4909 generic.go:356] "Generic (PLEG): container finished" podID="c931adf2-8c4f-4539-be64-6d5f17e08d96" containerID="90ad301661cdc992b54a633ca207d902b76f06543fad4e2796e1f209482e8b64" exitCode=0
Nov 03 13:18:07.630063 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.629932    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" event={"ID":"c931adf2-8c4f-4539-be64-6d5f17e08d96","Type":"ContainerDied","Data":"90ad301661cdc992b54a633ca207d902b76f06543fad4e2796e1f209482e8b64"}
Nov 03 13:18:07.724681 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.724656    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 13:18:07.808187 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.808164    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 13:18:07.886205 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.886118    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c931adf2-8c4f-4539-be64-6d5f17e08d96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931adf2-8c4f-4539-be64-6d5f17e08d96-plugins-config-volume\") pod \"c931adf2-8c4f-4539-be64-6d5f17e08d96\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") "
Nov 03 13:18:07.886470 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.886439    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c931adf2-8c4f-4539-be64-6d5f17e08d96-kube-api-access-5gkss\" (UniqueName: \"kubernetes.io/projected/c931adf2-8c4f-4539-be64-6d5f17e08d96-kube-api-access-5gkss\") pod \"c931adf2-8c4f-4539-be64-6d5f17e08d96\" (UID: \"c931adf2-8c4f-4539-be64-6d5f17e08d96\") "
Nov 03 13:18:07.886576 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.886516    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c931adf2-8c4f-4539-be64-6d5f17e08d96-plugins-config-volume" pod "c931adf2-8c4f-4539-be64-6d5f17e08d96" (UID: "c931adf2-8c4f-4539-be64-6d5f17e08d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:07.886965 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.886941    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931adf2-8c4f-4539-be64-6d5f17e08d96-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:07.888885 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.888861    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c931adf2-8c4f-4539-be64-6d5f17e08d96-kube-api-access-5gkss" pod "c931adf2-8c4f-4539-be64-6d5f17e08d96" (UID: "c931adf2-8c4f-4539-be64-6d5f17e08d96"). InnerVolumeSpecName "kube-api-access-5gkss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:07.987673 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.987649    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e837472-8f8f-4682-ad78-0f7779e164f7-kube-api-access-sm2bm\" (UniqueName: \"kubernetes.io/projected/1e837472-8f8f-4682-ad78-0f7779e164f7-kube-api-access-sm2bm\") pod \"1e837472-8f8f-4682-ad78-0f7779e164f7\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") "
Nov 03 13:18:07.987839 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.987794    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e837472-8f8f-4682-ad78-0f7779e164f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e837472-8f8f-4682-ad78-0f7779e164f7-plugins-config-volume\") pod \"1e837472-8f8f-4682-ad78-0f7779e164f7\" (UID: \"1e837472-8f8f-4682-ad78-0f7779e164f7\") "
Nov 03 13:18:07.988078 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.988064    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gkss\" (UniqueName: \"kubernetes.io/projected/c931adf2-8c4f-4539-be64-6d5f17e08d96-kube-api-access-5gkss\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:07.988179 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.988157    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e837472-8f8f-4682-ad78-0f7779e164f7-plugins-config-volume" pod "1e837472-8f8f-4682-ad78-0f7779e164f7" (UID: "1e837472-8f8f-4682-ad78-0f7779e164f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:07.989835 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:07.989816    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e837472-8f8f-4682-ad78-0f7779e164f7-kube-api-access-sm2bm" pod "1e837472-8f8f-4682-ad78-0f7779e164f7" (UID: "1e837472-8f8f-4682-ad78-0f7779e164f7"). InnerVolumeSpecName "kube-api-access-sm2bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:08.089244 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.089213    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2bm\" (UniqueName: \"kubernetes.io/projected/1e837472-8f8f-4682-ad78-0f7779e164f7-kube-api-access-sm2bm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:08.089244 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.089240    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e837472-8f8f-4682-ad78-0f7779e164f7-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:08.637093 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.637060    4909 generic.go:356] "Generic (PLEG): container finished" podID="1e837472-8f8f-4682-ad78-0f7779e164f7" containerID="cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64" exitCode=0
Nov 03 13:18:08.637571 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.637122    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"
Nov 03 13:18:08.637571 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.637137    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" event={"ID":"1e837472-8f8f-4682-ad78-0f7779e164f7","Type":"ContainerDied","Data":"cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64"}
Nov 03 13:18:08.637571 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.637218    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72" event={"ID":"1e837472-8f8f-4682-ad78-0f7779e164f7","Type":"ContainerDied","Data":"322d27fa5248cb07c21df63d5c476eb75f8a73c9e3a3ac234241983de49ae7c7"}
Nov 03 13:18:08.637571 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.637246    4909 scope.go:122] "RemoveContainer" containerID="cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64"
Nov 03 13:18:08.638996 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.638967    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m" event={"ID":"c931adf2-8c4f-4539-be64-6d5f17e08d96","Type":"ContainerDied","Data":"8790e627ac36a72e4f369eb8986d252e20321f0b9b211fa75f06ed522ecd30ac"}
Nov 03 13:18:08.639123 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.639000    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"
Nov 03 13:18:08.648485 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.648470    4909 scope.go:122] "RemoveContainer" containerID="cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64"
Nov 03 13:18:08.648784 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:08.648764    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64\": container with ID starting with cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64 not found: ID does not exist" containerID="cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64"
Nov 03 13:18:08.648853 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.648791    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64"} err="failed to get container status \"cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64\": rpc error: code = NotFound desc = could not find container \"cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64\": container with ID starting with cbc88ea123fe7ea8c06c63680833437d036eafe1558be5a42c22135b95ebbc64 not found: ID does not exist"
Nov 03 13:18:08.648853 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.648806    4909 scope.go:122] "RemoveContainer" containerID="90ad301661cdc992b54a633ca207d902b76f06543fad4e2796e1f209482e8b64"
Nov 03 13:18:08.790060 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.790004    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"]
Nov 03 13:18:08.794803 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.794780    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6fb3a8-74db876fd5-hsd8m"]
Nov 03 13:18:08.895685 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.895659    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"]
Nov 03 13:18:08.899915 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:08.899895    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6fb3a8-5b664cfff-qrb72"]
Nov 03 13:18:09.016721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:09.016680    4909 patch_prober.go:29] interesting pod/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:18:09.016864 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:09.016755    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:18:09.122400 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:09.122330    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e837472-8f8f-4682-ad78-0f7779e164f7" path="/var/lib/kubelet/pods/1e837472-8f8f-4682-ad78-0f7779e164f7/volumes"
Nov 03 13:18:09.122737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:09.122722    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c931adf2-8c4f-4539-be64-6d5f17e08d96" path="/var/lib/kubelet/pods/c931adf2-8c4f-4539-be64-6d5f17e08d96/volumes"
Nov 03 13:18:09.563642 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:09.563605    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:18:12.550480 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.550457    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 13:18:12.632226 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632157    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-podinfo\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632226 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632210    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-certs\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632231    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-data\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632255    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istiod-ca-cert\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632288    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-token\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-token\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632331    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-kube-api-access-ngcxl\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-kube-api-access-ngcxl\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632395 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632387    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-socket\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632426    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-credential-socket\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632466    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-envoy\") pod \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\" (UID: \"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6\") "
Nov 03 13:18:12.632678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632574    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-certs" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:12.632678 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632609    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-data" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:12.632966 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632728    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istiod-ca-cert" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:12.632966 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632934    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-socket" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:12.633089 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632987    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-credential-socket" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:12.633089 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.632990    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.633089 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.633025    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.633089 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.633068    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.634422 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.634399    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-podinfo" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:18:12.634524 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.634503    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-token" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:12.634925 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.634907    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-envoy" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:12.634963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.634935    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-kube-api-access-ngcxl" pod "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" (UID: "d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6"). InnerVolumeSpecName "kube-api-access-ngcxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:12.670211 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.670165    4909 generic.go:356] "Generic (PLEG): container finished" podID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerID="8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661" exitCode=0
Nov 03 13:18:12.670412 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.670354    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" event={"ID":"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6","Type":"ContainerDied","Data":"8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661"}
Nov 03 13:18:12.670412 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.670399    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx" event={"ID":"d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6","Type":"ContainerDied","Data":"6938525873ec4a179294ae1d9a597f5fabf820876b8cdb83347d32e8cfa55ed8"}
Nov 03 13:18:12.670542 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.670422    4909 scope.go:122] "RemoveContainer" containerID="8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661"
Nov 03 13:18:12.670542 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.670504    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"
Nov 03 13:18:12.693943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.693921    4909 scope.go:122] "RemoveContainer" containerID="963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94"
Nov 03 13:18:12.703792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.703771    4909 scope.go:122] "RemoveContainer" containerID="8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661"
Nov 03 13:18:12.704090 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:12.704065    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661\": container with ID starting with 8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661 not found: ID does not exist" containerID="8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661"
Nov 03 13:18:12.704170 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.704097    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661"} err="failed to get container status \"8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661\": rpc error: code = NotFound desc = could not find container \"8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661\": container with ID starting with 8193c42ee3b697a001c20d06e3bfe2f012bc30c4379f72dc2fce9c6341b7d661 not found: ID does not exist"
Nov 03 13:18:12.704170 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.704112    4909 scope.go:122] "RemoveContainer" containerID="963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94"
Nov 03 13:18:12.704345 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:12.704327    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94\": container with ID starting with 963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94 not found: ID does not exist" containerID="963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94"
Nov 03 13:18:12.704394 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.704350    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94"} err="failed to get container status \"963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94\": rpc error: code = NotFound desc = could not find container \"963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94\": container with ID starting with 963b71e7d613604cc30def48c8457fa2d755b6ff0aee8d2fa0d565c6f8c9ec94 not found: ID does not exist"
Nov 03 13:18:12.734053 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.734025    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.734162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.734055    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.734162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.734069    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngcxl\" (UniqueName: \"kubernetes.io/projected/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-kube-api-access-ngcxl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.734162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.734085    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.734162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.734099    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.734162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.734113    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:12.816135 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.816105    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"]
Nov 03 13:18:12.820807 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:12.820781    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-94e247-openshift-default-74979b6977-jl9bx"]
Nov 03 13:18:13.122494 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:13.122462    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" path="/var/lib/kubelet/pods/d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6/volumes"
Nov 03 13:18:14.484212 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:14.484175    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"]
Nov 03 13:18:14.595804 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:14.595771    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"]
Nov 03 13:18:17.938353 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:17.938319    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"]
Nov 03 13:18:17.984326 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:17.984297    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"]
Nov 03 13:18:19.484794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:19.484736    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" podUID="71ec8b30-32ec-407b-ad71-25dfdde68e79" containerName="payload-processing" containerID="cri-o://fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274" gracePeriod=25
Nov 03 13:18:19.597081 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:19.597042    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" podUID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" containerName="payload-pre-processing" containerID="cri-o://09d96dfe5b93f84c284a6badc64cca67f7141ca5c9185f5eecb1ec5e5bb312af" gracePeriod=25
Nov 03 13:18:27.591822 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:27.591784    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"]
Nov 03 13:18:27.592220 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:27.592004    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy" containerID="cri-o://87412adf4f4c7f39820e8237768b3fbc7e219e016b0f068d231524cc33ad0f5e" gracePeriod=30
Nov 03 13:18:28.795738 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:28.795676    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" podUID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:28.795738 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:28.795716    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" podUID="71ec8b30-32ec-407b-ad71-25dfdde68e79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:29.233213 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:29.233179    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"]
Nov 03 13:18:32.621992 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.621963    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:18:32.777973 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.777941    4909 generic.go:356] "Generic (PLEG): container finished" podID="396f2942-0950-46ad-812b-d506efc6bfbd" containerID="87412adf4f4c7f39820e8237768b3fbc7e219e016b0f068d231524cc33ad0f5e" exitCode=0
Nov 03 13:18:32.778136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.777985    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" event={"ID":"396f2942-0950-46ad-812b-d506efc6bfbd","Type":"ContainerDied","Data":"87412adf4f4c7f39820e8237768b3fbc7e219e016b0f068d231524cc33ad0f5e"}
Nov 03 13:18:32.778136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.778051    4909 scope.go:122] "RemoveContainer" containerID="bf75a593aa5f87ca9835edeb6baad3311b9567c931adab7b63caf4512bd5469d"
Nov 03 13:18:32.849726 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.849686    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 13:18:32.940820 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.940788    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-credential-socket\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.940842    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-certs\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.940899    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-data\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.940928    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396f2942-0950-46ad-812b-d506efc6bfbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396f2942-0950-46ad-812b-d506efc6bfbd-istiod-ca-cert\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.940983    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-kube-api-access-ls9mt\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-kube-api-access-ls9mt\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941047    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-istio-token\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-istio-token\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941069    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-credential-socket" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941087    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-envoy\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941232    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-certs" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941476    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396f2942-0950-46ad-812b-d506efc6bfbd-istiod-ca-cert" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941523    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-socket\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941625    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-data" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.941767    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/396f2942-0950-46ad-812b-d506efc6bfbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396f2942-0950-46ad-812b-d506efc6bfbd-istio-podinfo\") pod \"396f2942-0950-46ad-812b-d506efc6bfbd\" (UID: \"396f2942-0950-46ad-812b-d506efc6bfbd\") "
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.942159    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-socket" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.942184    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.942207    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396f2942-0950-46ad-812b-d506efc6bfbd-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.942223    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:32.946085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.942236    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:32.950723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.950536    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/396f2942-0950-46ad-812b-d506efc6bfbd-istio-podinfo" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:18:32.950723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.950661    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-istio-token" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:32.950937 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.950897    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-envoy" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:32.950937 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.950910    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-kube-api-access-ls9mt" pod "396f2942-0950-46ad-812b-d506efc6bfbd" (UID: "396f2942-0950-46ad-812b-d506efc6bfbd"). InnerVolumeSpecName "kube-api-access-ls9mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:32.959632 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:32.959600    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main" containerID="cri-o://3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359" gracePeriod=45
Nov 03 13:18:33.003324 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.003289    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" containerID="cri-o://bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e" gracePeriod=45
Nov 03 13:18:33.042891 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.042866    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396f2942-0950-46ad-812b-d506efc6bfbd-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.043000 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.042892    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9mt\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-kube-api-access-ls9mt\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.043000 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.042902    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396f2942-0950-46ad-812b-d506efc6bfbd-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.043000 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.042910    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.043000 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.042920    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396f2942-0950-46ad-812b-d506efc6bfbd-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.231485 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.231460    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 13:18:33.262409 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.262385    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 13:18:33.345291 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.345260    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-dshm\") pod \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") "
Nov 03 13:18:33.345683 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.345661    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tls-certs\") pod \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") "
Nov 03 13:18:33.345870 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.345845    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-model-cache\") pod \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.345895    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4120763b-a5bc-481d-a9ef-7d178c40b7b8-kube-api-access-2f6j2\" (UniqueName: \"kubernetes.io/projected/4120763b-a5bc-481d-a9ef-7d178c40b7b8-kube-api-access-2f6j2\") pod \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.345959    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-home\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-home\") pod \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.345993    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-dshm\") pod \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346024    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tmp-dir\") pod \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346068    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-model-cache\") pod \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346095    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-home\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-home\") pod \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346132    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-model-cache" pod "4120763b-a5bc-481d-a9ef-7d178c40b7b8" (UID: "4120763b-a5bc-481d-a9ef-7d178c40b7b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346143    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tls-certs\") pod \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346208    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f1faf3-c922-46e8-b9a9-c68465c47a93-kube-api-access-hl7w4\" (UniqueName: \"kubernetes.io/projected/c7f1faf3-c922-46e8-b9a9-c68465c47a93-kube-api-access-hl7w4\") pod \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\" (UID: \"c7f1faf3-c922-46e8-b9a9-c68465c47a93\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346311    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tmp-dir\") pod \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\" (UID: \"4120763b-a5bc-481d-a9ef-7d178c40b7b8\") "
Nov 03 13:18:33.346866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346622    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tmp-dir" pod "c7f1faf3-c922-46e8-b9a9-c68465c47a93" (UID: "c7f1faf3-c922-46e8-b9a9-c68465c47a93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.347613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346828    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-model-cache" pod "c7f1faf3-c922-46e8-b9a9-c68465c47a93" (UID: "c7f1faf3-c922-46e8-b9a9-c68465c47a93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.347613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.346888    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tmp-dir" pod "4120763b-a5bc-481d-a9ef-7d178c40b7b8" (UID: "4120763b-a5bc-481d-a9ef-7d178c40b7b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.347613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.347229    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.347613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.347255    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.347613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.347351    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-home" pod "c7f1faf3-c922-46e8-b9a9-c68465c47a93" (UID: "c7f1faf3-c922-46e8-b9a9-c68465c47a93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.347613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.347599    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-home" pod "4120763b-a5bc-481d-a9ef-7d178c40b7b8" (UID: "4120763b-a5bc-481d-a9ef-7d178c40b7b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.349536 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.349503    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tls-certs" pod "c7f1faf3-c922-46e8-b9a9-c68465c47a93" (UID: "c7f1faf3-c922-46e8-b9a9-c68465c47a93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:33.349888 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.349852    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tls-certs" pod "4120763b-a5bc-481d-a9ef-7d178c40b7b8" (UID: "4120763b-a5bc-481d-a9ef-7d178c40b7b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:33.350312 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.350288    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4120763b-a5bc-481d-a9ef-7d178c40b7b8-kube-api-access-2f6j2" pod "4120763b-a5bc-481d-a9ef-7d178c40b7b8" (UID: "4120763b-a5bc-481d-a9ef-7d178c40b7b8"). InnerVolumeSpecName "kube-api-access-2f6j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:33.350464 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.350446    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-dshm" pod "4120763b-a5bc-481d-a9ef-7d178c40b7b8" (UID: "4120763b-a5bc-481d-a9ef-7d178c40b7b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.350793 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.350768    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f1faf3-c922-46e8-b9a9-c68465c47a93-kube-api-access-hl7w4" pod "c7f1faf3-c922-46e8-b9a9-c68465c47a93" (UID: "c7f1faf3-c922-46e8-b9a9-c68465c47a93"). InnerVolumeSpecName "kube-api-access-hl7w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:33.350935 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.350902    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-dshm" pod "c7f1faf3-c922-46e8-b9a9-c68465c47a93" (UID: "c7f1faf3-c922-46e8-b9a9-c68465c47a93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.448843 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448807    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1faf3-c922-46e8-b9a9-c68465c47a93-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.448843 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448846    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl7w4\" (UniqueName: \"kubernetes.io/projected/c7f1faf3-c922-46e8-b9a9-c68465c47a93-kube-api-access-hl7w4\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448863    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448877    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448890    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120763b-a5bc-481d-a9ef-7d178c40b7b8-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448903    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f6j2\" (UniqueName: \"kubernetes.io/projected/4120763b-a5bc-481d-a9ef-7d178c40b7b8-kube-api-access-2f6j2\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448916    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448930    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448943    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f1faf3-c922-46e8-b9a9-c68465c47a93-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.449113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.448957    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4120763b-a5bc-481d-a9ef-7d178c40b7b8-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.784600 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.784563    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"
Nov 03 13:18:33.784600 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.784592    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g" event={"ID":"396f2942-0950-46ad-812b-d506efc6bfbd","Type":"ContainerDied","Data":"e651538e4689cfaa022131e079a8730667162ef91331bfc99d6ca7e3ff041684"}
Nov 03 13:18:33.785154 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.784655    4909 scope.go:122] "RemoveContainer" containerID="87412adf4f4c7f39820e8237768b3fbc7e219e016b0f068d231524cc33ad0f5e"
Nov 03 13:18:33.786068 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.786042    4909 generic.go:356] "Generic (PLEG): container finished" podID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerID="bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e" exitCode=0
Nov 03 13:18:33.786174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.786079    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" event={"ID":"c7f1faf3-c922-46e8-b9a9-c68465c47a93","Type":"ContainerDied","Data":"bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e"}
Nov 03 13:18:33.786174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.786134    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" event={"ID":"c7f1faf3-c922-46e8-b9a9-c68465c47a93","Type":"ContainerDied","Data":"d9ebf99ede9537fbb748ddbae05b6f2031ba78132bf54568c1e7c05a7c8b7ee7"}
Nov 03 13:18:33.786174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.786085    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"
Nov 03 13:18:33.788064 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.788041    4909 generic.go:356] "Generic (PLEG): container finished" podID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerID="3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359" exitCode=0
Nov 03 13:18:33.788156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.788071    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" event={"ID":"4120763b-a5bc-481d-a9ef-7d178c40b7b8","Type":"ContainerDied","Data":"3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359"}
Nov 03 13:18:33.788156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.788131    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j" event={"ID":"4120763b-a5bc-481d-a9ef-7d178c40b7b8","Type":"ContainerDied","Data":"80ec2e9cb774983ffd0fdbf93de54897ac137335a1eabe498a2d95ab0d8a9ba1"}
Nov 03 13:18:33.788156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.788089    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"
Nov 03 13:18:33.794682 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.794662    4909 scope.go:122] "RemoveContainer" containerID="bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e"
Nov 03 13:18:33.806284 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.806264    4909 scope.go:122] "RemoveContainer" containerID="bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e"
Nov 03 13:18:33.806578 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:33.806561    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e\": container with ID starting with bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e not found: ID does not exist" containerID="bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e"
Nov 03 13:18:33.806630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.806585    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e"} err="failed to get container status \"bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e\": rpc error: code = NotFound desc = could not find container \"bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e\": container with ID starting with bdace3e34bdf8d1a80a3f04315af66d8f518a83f8f289df205e4d7ccc30d211e not found: ID does not exist"
Nov 03 13:18:33.806630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.806602    4909 scope.go:122] "RemoveContainer" containerID="3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359"
Nov 03 13:18:33.814298 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.814277    4909 scope.go:122] "RemoveContainer" containerID="3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359"
Nov 03 13:18:33.814558 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:33.814538    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359\": container with ID starting with 3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359 not found: ID does not exist" containerID="3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359"
Nov 03 13:18:33.814609 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.814566    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359"} err="failed to get container status \"3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359\": rpc error: code = NotFound desc = could not find container \"3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359\": container with ID starting with 3ecdd1cbce5345451993c1e703ad49ce8c5a54319f1a28d0f406e7ffc2e02359 not found: ID does not exist"
Nov 03 13:18:33.937084 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.937030    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"]
Nov 03 13:18:33.982864 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:33.982838    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ab27b8-openshift-default-9784c7cf9-l872g"]
Nov 03 13:18:34.044906 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.044880    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"]
Nov 03 13:18:34.049490 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.049466    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-w1-5951bc-kserve-6c6f5b85f-hgq2j"]
Nov 03 13:18:34.150508 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.150484    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"]
Nov 03 13:18:34.154605 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.154582    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8"]
Nov 03 13:18:34.234413 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.234384    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5" podUID="2100fe6d-fc50-4311-abe3-0b140c4701ed" containerName="payload-pre-processing" containerID="cri-o://8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200" gracePeriod=25
Nov 03 13:18:34.248850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.248822    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/premium-w1-5951bc-kserve-84c4dddb68-mfwt8" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 03 13:18:34.496955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.496898    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 13:18:34.659149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.659115    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2100fe6d-fc50-4311-abe3-0b140c4701ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2100fe6d-fc50-4311-abe3-0b140c4701ed-plugins-config-volume\") pod \"2100fe6d-fc50-4311-abe3-0b140c4701ed\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") "
Nov 03 13:18:34.659350 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.659256    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2100fe6d-fc50-4311-abe3-0b140c4701ed-kube-api-access-wqqs2\" (UniqueName: \"kubernetes.io/projected/2100fe6d-fc50-4311-abe3-0b140c4701ed-kube-api-access-wqqs2\") pod \"2100fe6d-fc50-4311-abe3-0b140c4701ed\" (UID: \"2100fe6d-fc50-4311-abe3-0b140c4701ed\") "
Nov 03 13:18:34.659494 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.659471    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2100fe6d-fc50-4311-abe3-0b140c4701ed-plugins-config-volume" pod "2100fe6d-fc50-4311-abe3-0b140c4701ed" (UID: "2100fe6d-fc50-4311-abe3-0b140c4701ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:34.659585 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.659572    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2100fe6d-fc50-4311-abe3-0b140c4701ed-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:34.661730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.661689    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2100fe6d-fc50-4311-abe3-0b140c4701ed-kube-api-access-wqqs2" pod "2100fe6d-fc50-4311-abe3-0b140c4701ed" (UID: "2100fe6d-fc50-4311-abe3-0b140c4701ed"). InnerVolumeSpecName "kube-api-access-wqqs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:34.760566 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.760505    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqqs2\" (UniqueName: \"kubernetes.io/projected/2100fe6d-fc50-4311-abe3-0b140c4701ed-kube-api-access-wqqs2\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:34.794330 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.794304    4909 generic.go:356] "Generic (PLEG): container finished" podID="2100fe6d-fc50-4311-abe3-0b140c4701ed" containerID="8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200" exitCode=0
Nov 03 13:18:34.794731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.794353    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"
Nov 03 13:18:34.794731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.794393    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5" event={"ID":"2100fe6d-fc50-4311-abe3-0b140c4701ed","Type":"ContainerDied","Data":"8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200"}
Nov 03 13:18:34.794731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.794454    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5" event={"ID":"2100fe6d-fc50-4311-abe3-0b140c4701ed","Type":"ContainerDied","Data":"cdd3778aefd3614f1ac3950a89de27eca1e074048d2b1180b33eb78481fdc79f"}
Nov 03 13:18:34.794731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.794487    4909 scope.go:122] "RemoveContainer" containerID="8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200"
Nov 03 13:18:34.803406 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.803389    4909 scope.go:122] "RemoveContainer" containerID="8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200"
Nov 03 13:18:34.803681 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:34.803661    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200\": container with ID starting with 8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200 not found: ID does not exist" containerID="8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200"
Nov 03 13:18:34.803811 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.803690    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200"} err="failed to get container status \"8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200\": rpc error: code = NotFound desc = could not find container \"8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200\": container with ID starting with 8bdbc79cda3793f527a51c57e17a392a29349e4297c43b0d772ad9d413ff1200 not found: ID does not exist"
Nov 03 13:18:34.946109 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.946081    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"]
Nov 03 13:18:34.950624 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:34.950601    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5951bc-6ff6844bd7-cfqg5"]
Nov 03 13:18:35.121817 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:35.121752    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2100fe6d-fc50-4311-abe3-0b140c4701ed" path="/var/lib/kubelet/pods/2100fe6d-fc50-4311-abe3-0b140c4701ed/volumes"
Nov 03 13:18:35.122197 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:35.122184    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" path="/var/lib/kubelet/pods/396f2942-0950-46ad-812b-d506efc6bfbd/volumes"
Nov 03 13:18:35.122558 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:35.122547    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" path="/var/lib/kubelet/pods/4120763b-a5bc-481d-a9ef-7d178c40b7b8/volumes"
Nov 03 13:18:35.123222 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:35.123208    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" path="/var/lib/kubelet/pods/c7f1faf3-c922-46e8-b9a9-c68465c47a93/volumes"
Nov 03 13:18:35.821012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:35.820983    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"]
Nov 03 13:18:38.795435 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:38.795400    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" podUID="71ec8b30-32ec-407b-ad71-25dfdde68e79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:38.795870 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:38.795401    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" podUID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:39.563425 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:39.563389    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:18:39.740685 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.740618    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 13:18:39.824746 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.824717    4909 generic.go:356] "Generic (PLEG): container finished" podID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" containerID="09d96dfe5b93f84c284a6badc64cca67f7141ca5c9185f5eecb1ec5e5bb312af" exitCode=0
Nov 03 13:18:39.825186 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.824798    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" event={"ID":"c8c320c1-862e-4421-bd97-f3e64bd88d4d","Type":"ContainerDied","Data":"09d96dfe5b93f84c284a6badc64cca67f7141ca5c9185f5eecb1ec5e5bb312af"}
Nov 03 13:18:39.826225 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.826199    4909 generic.go:356] "Generic (PLEG): container finished" podID="71ec8b30-32ec-407b-ad71-25dfdde68e79" containerID="fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274" exitCode=0
Nov 03 13:18:39.826378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.826261    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"
Nov 03 13:18:39.826378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.826262    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" event={"ID":"71ec8b30-32ec-407b-ad71-25dfdde68e79","Type":"ContainerDied","Data":"fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274"}
Nov 03 13:18:39.826493 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.826412    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb" event={"ID":"71ec8b30-32ec-407b-ad71-25dfdde68e79","Type":"ContainerDied","Data":"c5e7b75ddafef2c7bb94b9266bfed1d7ab0d289b917b70e5e95a3684920b51ae"}
Nov 03 13:18:39.826493 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.826443    4909 scope.go:122] "RemoveContainer" containerID="fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274"
Nov 03 13:18:39.841554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.841535    4909 scope.go:122] "RemoveContainer" containerID="fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274"
Nov 03 13:18:39.841837 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:18:39.841817    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274\": container with ID starting with fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274 not found: ID does not exist" containerID="fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274"
Nov 03 13:18:39.841925 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.841848    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274"} err="failed to get container status \"fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274\": rpc error: code = NotFound desc = could not find container \"fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274\": container with ID starting with fa509e2cf81658fa9089187499f797ccbeec88e04da9d050bca70ba2d8d0b274 not found: ID does not exist"
Nov 03 13:18:39.844109 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.844092    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 13:18:39.903551 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.903527    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ec8b30-32ec-407b-ad71-25dfdde68e79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec8b30-32ec-407b-ad71-25dfdde68e79-plugins-config-volume\") pod \"71ec8b30-32ec-407b-ad71-25dfdde68e79\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") "
Nov 03 13:18:39.903745 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.903720    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ec8b30-32ec-407b-ad71-25dfdde68e79-kube-api-access-dqn4h\" (UniqueName: \"kubernetes.io/projected/71ec8b30-32ec-407b-ad71-25dfdde68e79-kube-api-access-dqn4h\") pod \"71ec8b30-32ec-407b-ad71-25dfdde68e79\" (UID: \"71ec8b30-32ec-407b-ad71-25dfdde68e79\") "
Nov 03 13:18:39.903969 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.903948    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ec8b30-32ec-407b-ad71-25dfdde68e79-plugins-config-volume" pod "71ec8b30-32ec-407b-ad71-25dfdde68e79" (UID: "71ec8b30-32ec-407b-ad71-25dfdde68e79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:39.904038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.904023    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec8b30-32ec-407b-ad71-25dfdde68e79-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:39.905640 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:39.905620    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ec8b30-32ec-407b-ad71-25dfdde68e79-kube-api-access-dqn4h" pod "71ec8b30-32ec-407b-ad71-25dfdde68e79" (UID: "71ec8b30-32ec-407b-ad71-25dfdde68e79"). InnerVolumeSpecName "kube-api-access-dqn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:40.004837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.004769    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c320c1-862e-4421-bd97-f3e64bd88d4d-kube-api-access-zplbj\" (UniqueName: \"kubernetes.io/projected/c8c320c1-862e-4421-bd97-f3e64bd88d4d-kube-api-access-zplbj\") pod \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") "
Nov 03 13:18:40.004963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.004853    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c320c1-862e-4421-bd97-f3e64bd88d4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c320c1-862e-4421-bd97-f3e64bd88d4d-plugins-config-volume\") pod \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\" (UID: \"c8c320c1-862e-4421-bd97-f3e64bd88d4d\") "
Nov 03 13:18:40.005062 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.005049    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqn4h\" (UniqueName: \"kubernetes.io/projected/71ec8b30-32ec-407b-ad71-25dfdde68e79-kube-api-access-dqn4h\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:40.005218 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.005194    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c320c1-862e-4421-bd97-f3e64bd88d4d-plugins-config-volume" pod "c8c320c1-862e-4421-bd97-f3e64bd88d4d" (UID: "c8c320c1-862e-4421-bd97-f3e64bd88d4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:40.007031 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.007012    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c320c1-862e-4421-bd97-f3e64bd88d4d-kube-api-access-zplbj" pod "c8c320c1-862e-4421-bd97-f3e64bd88d4d" (UID: "c8c320c1-862e-4421-bd97-f3e64bd88d4d"). InnerVolumeSpecName "kube-api-access-zplbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:40.106048 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.106018    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c320c1-862e-4421-bd97-f3e64bd88d4d-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:40.106048 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.106044    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplbj\" (UniqueName: \"kubernetes.io/projected/c8c320c1-862e-4421-bd97-f3e64bd88d4d-kube-api-access-zplbj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:40.237521 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.237490    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"]
Nov 03 13:18:40.250847 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.250821    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ab27b8-6b4854977c-85rbb"]
Nov 03 13:18:40.821606 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.821527    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" podUID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" containerName="payload-pre-processing" containerID="cri-o://5f8fac862e9bf0009b9f0932fa4631ca65a4e9ee3178af4c9ee13f11c4a91a3f" gracePeriod=25
Nov 03 13:18:40.833156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.833131    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2" event={"ID":"c8c320c1-862e-4421-bd97-f3e64bd88d4d","Type":"ContainerDied","Data":"7bc75e86df09f85a940e86fddecb3d83b797619bf02fc4fe6254172f6d09032b"}
Nov 03 13:18:40.833156 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.833152    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"
Nov 03 13:18:40.833615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.833184    4909 scope.go:122] "RemoveContainer" containerID="09d96dfe5b93f84c284a6badc64cca67f7141ca5c9185f5eecb1ec5e5bb312af"
Nov 03 13:18:40.988965 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.988934    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"]
Nov 03 13:18:40.993625 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:40.993605    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ab27b8-9f5dcf9ff-dmtx2"]
Nov 03 13:18:41.121512 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:41.121429    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ec8b30-32ec-407b-ad71-25dfdde68e79" path="/var/lib/kubelet/pods/71ec8b30-32ec-407b-ad71-25dfdde68e79/volumes"
Nov 03 13:18:41.121927 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:41.121905    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" path="/var/lib/kubelet/pods/c8c320c1-862e-4421-bd97-f3e64bd88d4d/volumes"
Nov 03 13:18:46.857795 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:46.857754    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" podUID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:54.203481 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203448    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"]
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203861    4909 cpu_manager.go:414] "containerMap: removing container" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203873    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203883    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203888    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203898    4909 cpu_manager.go:414] "containerMap: removing container" podUID="53e5714c-34d7-4250-8aad-724b2915baf7" containerName="maas-api"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203906    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e5714c-34d7-4250-8aad-724b2915baf7" containerName="maas-api"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203916    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203922    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.203928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203931    4909 cpu_manager.go:414] "containerMap: removing container" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203936    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203946    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203951    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203958    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4958a745-d677-4897-8d31-cf29f2a67b9c" containerName="payload-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203962    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4958a745-d677-4897-8d31-cf29f2a67b9c" containerName="payload-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203968    4909 cpu_manager.go:414] "containerMap: removing container" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203973    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203979    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203983    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203989    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203993    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.203998    4909 cpu_manager.go:414] "containerMap: removing container" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204002    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204008    4909 cpu_manager.go:414] "containerMap: removing container" podUID="8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" containerName="maas-api"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204013    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" containerName="maas-api"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204022    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204027    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204035    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204040    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204045    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204049    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204057    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204062    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204070    4909 cpu_manager.go:414] "containerMap: removing container" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204074    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204082    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2100fe6d-fc50-4311-abe3-0b140c4701ed" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204086    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2100fe6d-fc50-4311-abe3-0b140c4701ed" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204094    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c931adf2-8c4f-4539-be64-6d5f17e08d96" containerName="payload-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204098    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c931adf2-8c4f-4539-be64-6d5f17e08d96" containerName="payload-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204106    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204111    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204117    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4b1569f9-71ca-42c0-b16d-4d59b9cd7764" containerName="main"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204121    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1569f9-71ca-42c0-b16d-4d59b9cd7764" containerName="main"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204128    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ded63b27-8f76-41d1-a992-38982f6a69b4" containerName="payload-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204132    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded63b27-8f76-41d1-a992-38982f6a69b4" containerName="payload-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204137    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204142    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204147    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204152    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204161    4909 cpu_manager.go:414] "containerMap: removing container" podUID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204165    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204172    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204176    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204181    4909 cpu_manager.go:414] "containerMap: removing container" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204185    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204192    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204197    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204203    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204207    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204213    4909 cpu_manager.go:414] "containerMap: removing container" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204218    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204224    4909 cpu_manager.go:414] "containerMap: removing container" podUID="955970d0-cd2b-46b5-8d24-4cf06cf33989" containerName="main"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204229    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955970d0-cd2b-46b5-8d24-4cf06cf33989" containerName="main"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204236    4909 cpu_manager.go:414] "containerMap: removing container" podUID="cc822e2b-def1-4eb9-a22b-f03dc395205c" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204240    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc822e2b-def1-4eb9-a22b-f03dc395205c" containerName="payload-pre-processing"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204247    4909 cpu_manager.go:414] "containerMap: removing container" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:18:54.204221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204252    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204258    4909 cpu_manager.go:414] "containerMap: removing container" podUID="ef3638d1-389f-41b8-8fa1-0d3053a39f63" containerName="revoke-keys"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204263    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3638d1-389f-41b8-8fa1-0d3053a39f63" containerName="revoke-keys"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204270    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204274    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204280    4909 cpu_manager.go:414] "containerMap: removing container" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204284    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204290    4909 cpu_manager.go:414] "containerMap: removing container" podUID="101f028b-4d7a-4293-a325-905a16670ba4" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204295    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101f028b-4d7a-4293-a325-905a16670ba4" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204300    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204305    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204312    4909 cpu_manager.go:414] "containerMap: removing container" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204316    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204321    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d11e72b2-5094-45a0-b147-7d86098e89c3" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204325    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11e72b2-5094-45a0-b147-7d86098e89c3" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204334    4909 cpu_manager.go:414] "containerMap: removing container" podUID="1e837472-8f8f-4682-ad78-0f7779e164f7" containerName="payload-pre-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204338    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e837472-8f8f-4682-ad78-0f7779e164f7" containerName="payload-pre-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204345    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204349    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204360    4909 cpu_manager.go:414] "containerMap: removing container" podUID="71ec8b30-32ec-407b-ad71-25dfdde68e79" containerName="payload-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204364    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ec8b30-32ec-407b-ad71-25dfdde68e79" containerName="payload-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204371    4909 cpu_manager.go:414] "containerMap: removing container" podUID="71fcf678-1da0-47bf-b12d-714d130fc6ea" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204375    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fcf678-1da0-47bf-b12d-714d130fc6ea" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204431    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204440    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1569f9-71ca-42c0-b16d-4d59b9cd7764" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204447    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded63b27-8f76-41d1-a992-38982f6a69b4" containerName="payload-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204453    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204458    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204464    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="0508f102-e252-43f4-a8a9-58fe6b0a06e3" containerName="payload-pre-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204468    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204473    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204477    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204483    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204488    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3957cb6-a26a-45f6-8b93-5abc02b422ec" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204493    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204499    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="955970d0-cd2b-46b5-8d24-4cf06cf33989" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204504    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc822e2b-def1-4eb9-a22b-f03dc395205c" containerName="payload-pre-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204510    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204514    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3638d1-389f-41b8-8fa1-0d3053a39f63" containerName="revoke-keys"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204519    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f731ab06-b572-43e3-9fef-972fed967e57" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204524    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f1faf3-c922-46e8-b9a9-c68465c47a93" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204530    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="101f028b-4d7a-4293-a325-905a16670ba4" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204534    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204539    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4120763b-a5bc-481d-a9ef-7d178c40b7b8" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204544    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204548    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11e72b2-5094-45a0-b147-7d86098e89c3" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204553    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e837472-8f8f-4682-ad78-0f7779e164f7" containerName="payload-pre-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204558    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e314af-4ea2-419e-879c-eaa78cedb602" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204563    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204569    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ec8b30-32ec-407b-ad71-25dfdde68e79" containerName="payload-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204573    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fcf678-1da0-47bf-b12d-714d130fc6ea" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204578    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e5714c-34d7-4250-8aad-724b2915baf7" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204582    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204587    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="589f3710-4603-4806-916d-d3e41a58c0f6" containerName="main"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204593    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62df49b-98a6-46dd-9f16-64466785b187" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204599    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="4958a745-d677-4897-8d31-cf29f2a67b9c" containerName="payload-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204604    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204609    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4676bc-f7ae-4b2c-9d4b-39b10bb116d5" containerName="maas-api"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204614    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2059f309-d7d6-4d00-b3cb-4d554b564068" containerName="istio-proxy"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204619    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d788795-759c-423b-b487-f8e5e44028f0" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204625    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c320c1-862e-4421-bd97-f3e64bd88d4d" containerName="payload-pre-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204631    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204635    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2100fe6d-fc50-4311-abe3-0b140c4701ed" containerName="payload-pre-processing"
Nov 03 13:18:54.206198 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.204640    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="c931adf2-8c4f-4539-be64-6d5f17e08d96" containerName="payload-processing"
Nov 03 13:18:54.209626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.209609    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.228878 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.228850    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.228991 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.228894    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.228991 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.228920    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.229087 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.229048    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qmk\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-kube-api-access-q6qmk\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.229126 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.229084    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.229126 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.229113    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.229204 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.229136    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.229248 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.229207    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.229292 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.229246    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.306413 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.306388    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5951bc-openshift-default-dockercfg-bp9bz\""
Nov 03 13:18:54.308249 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.308228    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"]
Nov 03 13:18:54.330259 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330236    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330278    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330299    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330323    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qmk\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-kube-api-access-q6qmk\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330345    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330378 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330368    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330668 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330390    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330668 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330418    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330668 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330441    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330942 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330669    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330942 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330778    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330942 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330848    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.330942 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.330881    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.331097 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.331067    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.332670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.332643    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.332976 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.332959    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.383787 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.383758    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.403940 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.403908    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qmk\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-kube-api-access-q6qmk\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.523142 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.523073    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:54.935694 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.935666    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"]
Nov 03 13:18:54.937762 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:18:54.937731    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2367c2e_9dea_4a04_809c_922ded3e2ea3.slice/crio-9862eff2ad121377dcec2fe3f680dd2a6d2ebf1941a7ce00d1539ec39ba0d60e WatchSource:0}: Error finding container 9862eff2ad121377dcec2fe3f680dd2a6d2ebf1941a7ce00d1539ec39ba0d60e: Status 404 returned error can't find the container with id 9862eff2ad121377dcec2fe3f680dd2a6d2ebf1941a7ce00d1539ec39ba0d60e
Nov 03 13:18:54.940279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.940238    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:18:54.940390 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.940314    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:18:54.940390 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:54.940350    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:18:55.927085 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:55.927048    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" event={"ID":"a2367c2e-9dea-4a04-809c-922ded3e2ea3","Type":"ContainerStarted","Data":"b352aad1590b51bb020cf70b5a039b27e1ba260eeeb51efb9ac4946e6af6d233"}
Nov 03 13:18:55.927593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:55.927100    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" event={"ID":"a2367c2e-9dea-4a04-809c-922ded3e2ea3","Type":"ContainerStarted","Data":"9862eff2ad121377dcec2fe3f680dd2a6d2ebf1941a7ce00d1539ec39ba0d60e"}
Nov 03 13:18:56.524053 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:56.524018    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:18:56.525235 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:56.525203    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 03 13:18:56.525364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:56.525257    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 03 13:18:56.863251 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:56.863159    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" podUID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:57.523870 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:57.523833    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 03 13:18:57.524261 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:57.523889    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 03 13:18:58.274052 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:58.273992    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" podStartSLOduration=4.273970347 podStartE2EDuration="4.273970347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:56.057967176 +0000 UTC m=+5285.070433843" watchObservedRunningTime="2026-11-03 13:18:58.273970347 +0000 UTC m=+5287.286437008"
Nov 03 13:18:58.289225 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:58.289191    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"]
Nov 03 13:18:58.526364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:58.525860    4909 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 03 13:18:58.526364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:58.525927    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 03 13:18:58.962721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:18:58.960741    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerName="istio-proxy" containerID="cri-o://b352aad1590b51bb020cf70b5a039b27e1ba260eeeb51efb9ac4946e6af6d233" gracePeriod=30
Nov 03 13:19:00.974623 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:00.974585    4909 generic.go:356] "Generic (PLEG): container finished" podID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" containerID="5f8fac862e9bf0009b9f0932fa4631ca65a4e9ee3178af4c9ee13f11c4a91a3f" exitCode=0
Nov 03 13:19:00.975172 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:00.974668    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" event={"ID":"92f9d76a-293c-4eed-bca2-8bbaae21de1b","Type":"ContainerDied","Data":"5f8fac862e9bf0009b9f0932fa4631ca65a4e9ee3178af4c9ee13f11c4a91a3f"}
Nov 03 13:19:00.991305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:00.991261    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"]
Nov 03 13:19:00.991562 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:00.991533    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy" containerID="cri-o://cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68" gracePeriod=30
Nov 03 13:19:01.121477 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.121445    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 13:19:01.213206 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.213169    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f9d76a-293c-4eed-bca2-8bbaae21de1b-kube-api-access-6pzbh\" (UniqueName: \"kubernetes.io/projected/92f9d76a-293c-4eed-bca2-8bbaae21de1b-kube-api-access-6pzbh\") pod \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") "
Nov 03 13:19:01.213403 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.213240    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f9d76a-293c-4eed-bca2-8bbaae21de1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f9d76a-293c-4eed-bca2-8bbaae21de1b-plugins-config-volume\") pod \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\" (UID: \"92f9d76a-293c-4eed-bca2-8bbaae21de1b\") "
Nov 03 13:19:01.213629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.213594    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f9d76a-293c-4eed-bca2-8bbaae21de1b-plugins-config-volume" pod "92f9d76a-293c-4eed-bca2-8bbaae21de1b" (UID: "92f9d76a-293c-4eed-bca2-8bbaae21de1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:01.215393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.215373    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f9d76a-293c-4eed-bca2-8bbaae21de1b-kube-api-access-6pzbh" pod "92f9d76a-293c-4eed-bca2-8bbaae21de1b" (UID: "92f9d76a-293c-4eed-bca2-8bbaae21de1b"). InnerVolumeSpecName "kube-api-access-6pzbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:01.313946 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.313909    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzbh\" (UniqueName: \"kubernetes.io/projected/92f9d76a-293c-4eed-bca2-8bbaae21de1b-kube-api-access-6pzbh\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:01.313946 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.313940    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f9d76a-293c-4eed-bca2-8bbaae21de1b-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:01.935112 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.935076    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:19:01.980886 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.980856    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"
Nov 03 13:19:01.981364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.980892    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw" event={"ID":"92f9d76a-293c-4eed-bca2-8bbaae21de1b","Type":"ContainerDied","Data":"d5bae86649923d8dc67cb651fdc641b44b0b472bf71ed30341c83276ca196db1"}
Nov 03 13:19:01.981364 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:01.980951    4909 scope.go:122] "RemoveContainer" containerID="5f8fac862e9bf0009b9f0932fa4631ca65a4e9ee3178af4c9ee13f11c4a91a3f"
Nov 03 13:19:02.137066 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:02.137031    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"]
Nov 03 13:19:02.141352 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:02.141324    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-743d9b-597bd7464c-qdbgw"]
Nov 03 13:19:03.125488 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:03.125457    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" path="/var/lib/kubelet/pods/92f9d76a-293c-4eed-bca2-8bbaae21de1b/volumes"
Nov 03 13:19:06.241164 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.241140    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 13:19:06.360653 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360585    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33323943-1d1c-4ad0-aa72-c73b1e72758a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33323943-1d1c-4ad0-aa72-c73b1e72758a-istiod-ca-cert\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.360810 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360666    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-envoy\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.360810 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360778    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-socket\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.360945 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360818    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-data\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.360945 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360868    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-podinfo\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.360945 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360907    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-certs\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.360945 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360906    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33323943-1d1c-4ad0-aa72-c73b1e72758a-istiod-ca-cert" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:06.361160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360944    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-kube-api-access-x2hg6\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-kube-api-access-x2hg6\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.361160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.360979    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-credential-socket\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.361160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361013    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-token\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-token\") pod \"33323943-1d1c-4ad0-aa72-c73b1e72758a\" (UID: \"33323943-1d1c-4ad0-aa72-c73b1e72758a\") "
Nov 03 13:19:06.361160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361024    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-socket" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:06.361160 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361136    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-data" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:06.361440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361238    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-certs" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:06.361440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361264    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-credential-socket" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:06.361440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361385    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.361440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361406    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.361440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361421    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.361440 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361435    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.361812 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.361448    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33323943-1d1c-4ad0-aa72-c73b1e72758a-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.363068 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.363038    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-envoy" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:06.363169 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.363071    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-podinfo" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:19:06.363169 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.363107    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-token" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:06.363522 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.363504    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-kube-api-access-x2hg6" pod "33323943-1d1c-4ad0-aa72-c73b1e72758a" (UID: "33323943-1d1c-4ad0-aa72-c73b1e72758a"). InnerVolumeSpecName "kube-api-access-x2hg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:06.462819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.462794    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.462941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.462819    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hg6\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-kube-api-access-x2hg6\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.462941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.462835    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.462941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:06.462847    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33323943-1d1c-4ad0-aa72-c73b1e72758a-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.012994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.012964    4909 generic.go:356] "Generic (PLEG): container finished" podID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerID="b352aad1590b51bb020cf70b5a039b27e1ba260eeeb51efb9ac4946e6af6d233" exitCode=0
Nov 03 13:19:07.013157 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.013035    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" event={"ID":"a2367c2e-9dea-4a04-809c-922ded3e2ea3","Type":"ContainerDied","Data":"b352aad1590b51bb020cf70b5a039b27e1ba260eeeb51efb9ac4946e6af6d233"}
Nov 03 13:19:07.014506 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.014483    4909 generic.go:356] "Generic (PLEG): container finished" podID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerID="cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68" exitCode=0
Nov 03 13:19:07.014647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.014520    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" event={"ID":"33323943-1d1c-4ad0-aa72-c73b1e72758a","Type":"ContainerDied","Data":"cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68"}
Nov 03 13:19:07.014647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.014550    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"
Nov 03 13:19:07.014647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.014566    4909 scope.go:122] "RemoveContainer" containerID="cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68"
Nov 03 13:19:07.014845 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.014555    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9" event={"ID":"33323943-1d1c-4ad0-aa72-c73b1e72758a","Type":"ContainerDied","Data":"e53463e0201ec630cd125ca7af48d85ebf6e13d99d829e37cac8fbba00c0c587"}
Nov 03 13:19:07.029917 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.029897    4909 scope.go:122] "RemoveContainer" containerID="cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68"
Nov 03 13:19:07.030171 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:19:07.030152    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68\": container with ID starting with cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68 not found: ID does not exist" containerID="cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68"
Nov 03 13:19:07.030228 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.030180    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68"} err="failed to get container status \"cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68\": rpc error: code = NotFound desc = could not find container \"cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68\": container with ID starting with cbea320886e0738abb7b8f3c0f27d17760b2fadec7d2fc2b96fc85344e7a1e68 not found: ID does not exist"
Nov 03 13:19:07.117799 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.117775    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:19:07.167679 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.167655    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"]
Nov 03 13:19:07.168692 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168671    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istiod-ca-cert\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.168784 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168762    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-socket\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.168852 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168804    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-envoy\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.168852 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168823    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-certs\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.168994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168873    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-kube-api-access-q6qmk\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-kube-api-access-q6qmk\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.168994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168909    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-credential-socket\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.168994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168948    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-data\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.168994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.168984    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-token\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-token\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.169229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169057    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-podinfo\") pod \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\" (UID: \"a2367c2e-9dea-4a04-809c-922ded3e2ea3\") "
Nov 03 13:19:07.169229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169104    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-socket" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:07.169229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169098    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istiod-ca-cert" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:07.169229 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169187    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-certs" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:07.169500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169348    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-credential-socket" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:07.169500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169364    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-data" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:07.169500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169472    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.169500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169493    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.169722 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169509    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.169722 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169524    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.169722 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.169538    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.171071 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.171047    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-envoy" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:07.171305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.171278    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-token" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:07.171402 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.171370    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-podinfo" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:19:07.171402 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.171391    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-kube-api-access-q6qmk" pod "a2367c2e-9dea-4a04-809c-922ded3e2ea3" (UID: "a2367c2e-9dea-4a04-809c-922ded3e2ea3"). InnerVolumeSpecName "kube-api-access-q6qmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:07.213305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.213244    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zjqz9"]
Nov 03 13:19:07.270116 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.270090    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.270116 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.270113    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.270442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.270128    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qmk\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-kube-api-access-q6qmk\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:07.270442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:07.270141    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2367c2e-9dea-4a04-809c-922ded3e2ea3-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:08.021914 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:08.021881    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"
Nov 03 13:19:08.021914 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:08.021903    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk" event={"ID":"a2367c2e-9dea-4a04-809c-922ded3e2ea3","Type":"ContainerDied","Data":"9862eff2ad121377dcec2fe3f680dd2a6d2ebf1941a7ce00d1539ec39ba0d60e"}
Nov 03 13:19:08.022166 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:08.021967    4909 scope.go:122] "RemoveContainer" containerID="b352aad1590b51bb020cf70b5a039b27e1ba260eeeb51efb9ac4946e6af6d233"
Nov 03 13:19:08.171776 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:08.171746    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"]
Nov 03 13:19:08.217637 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:08.217608    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-28bhk"]
Nov 03 13:19:09.121834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:09.121801    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" path="/var/lib/kubelet/pods/33323943-1d1c-4ad0-aa72-c73b1e72758a/volumes"
Nov 03 13:19:09.122298 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:09.122283    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" path="/var/lib/kubelet/pods/a2367c2e-9dea-4a04-809c-922ded3e2ea3/volumes"
Nov 03 13:19:09.563142 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:19:09.563111    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:19:26.731145 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:26.731104    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:19:31.327941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.327908    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"]
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328266    4909 cpu_manager.go:414] "containerMap: removing container" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy"
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328277    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f48ca3-7eb5-4e51-bf2c-d8f63dc44cb6" containerName="istio-proxy"
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328294    4909 cpu_manager.go:414] "containerMap: removing container" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy"
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328299    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy"
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328309    4909 cpu_manager.go:414] "containerMap: removing container" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328314    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328320    4909 cpu_manager.go:414] "containerMap: removing container" podUID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" containerName="payload-pre-processing"
Nov 03 13:19:31.328327 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328326    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" containerName="payload-pre-processing"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328337    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328343    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328355    4909 cpu_manager.go:414] "containerMap: removing container" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerName="istio-proxy"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328360    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerName="istio-proxy"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328416    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f9d76a-293c-4eed-bca2-8bbaae21de1b" containerName="payload-pre-processing"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328423    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7c238c-75f3-4f39-b899-a46a57487fcc" containerName="cleanup"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328432    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="536be7e2-613f-49d1-b30a-bb3cd6228494" containerName="cleanup"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328446    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0e5bfe-5ef5-4b68-9734-9b15c6c9e629" containerName="istio-proxy"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328454    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2367c2e-9dea-4a04-809c-922ded3e2ea3" containerName="istio-proxy"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328459    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8698500-d104-43a2-bb62-aae06cc3a4df" containerName="istio-proxy"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328467    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="396f2942-0950-46ad-812b-d506efc6bfbd" containerName="istio-proxy"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328473    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a8a03-06a4-49f2-8137-d49930e9e0ba" containerName="cleanup"
Nov 03 13:19:31.328595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.328481    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="33323943-1d1c-4ad0-aa72-c73b1e72758a" containerName="istio-proxy"
Nov 03 13:19:31.332953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.332931    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.381406 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.381380    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9893f0-openshift-default-dockercfg-ptlmk\""
Nov 03 13:19:31.429560 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.429535    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"]
Nov 03 13:19:31.480824 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.480787    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.480955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.480831    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.480955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.480852    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npnbw\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-kube-api-access-npnbw\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.480955 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.480927    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.481172 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.481030    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.481172 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.481076    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.481285 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.481191    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.481285 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.481224    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.481285 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.481253    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582061 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.581992    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582061 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582032    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npnbw\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-kube-api-access-npnbw\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582061 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582050    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582314 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582174    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582314 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582219    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582314 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582299    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582483 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582329    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582483 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582358    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582666 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582613    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582666 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582622    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.582887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582713    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.583011 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.582988    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.583093 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.583076    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.583093 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.583077    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.584797 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.584778    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.584797 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.584786    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.635120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.635094    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.635359 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.635324    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npnbw\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-kube-api-access-npnbw\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:31.644974 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:31.644947    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:32.060691 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.060664    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"]
Nov 03 13:19:32.062912 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:19:32.062880    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7eb47df_a81c_4419_8787_a3dfc6b7dd8b.slice/crio-15f578bf3d8fb085a5d82e684970797be8f00c1e9117b4b6561da14475bb7f47 WatchSource:0}: Error finding container 15f578bf3d8fb085a5d82e684970797be8f00c1e9117b4b6561da14475bb7f47: Status 404 returned error can't find the container with id 15f578bf3d8fb085a5d82e684970797be8f00c1e9117b4b6561da14475bb7f47
Nov 03 13:19:32.065038 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.065006    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:19:32.065104 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.065074    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:19:32.065141 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.065104    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:19:32.129482 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.129447    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" event={"ID":"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b","Type":"ContainerStarted","Data":"8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2"}
Nov 03 13:19:32.129575 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.129508    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" event={"ID":"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b","Type":"ContainerStarted","Data":"15f578bf3d8fb085a5d82e684970797be8f00c1e9117b4b6561da14475bb7f47"}
Nov 03 13:19:32.251016 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.250959    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" podStartSLOduration=1.250943519 podStartE2EDuration="1.250943519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:32.249482658 +0000 UTC m=+5321.261949354" watchObservedRunningTime="2026-11-03 13:19:32.250943519 +0000 UTC m=+5321.263410182"
Nov 03 13:19:32.655723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:32.648808    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:33.656722 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:33.653559    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 13:19:33.656722 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:33.653624    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" podUID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 13:19:34.596002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:34.595968    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"]
Nov 03 13:19:34.649121 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:34.649085    4909 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 13:19:34.649381 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:34.649353    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" podUID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 13:19:35.167087 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:35.167048    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" podUID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" containerName="istio-proxy" containerID="cri-o://8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2" gracePeriod=30
Nov 03 13:19:37.599869 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:37.599837    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xj776_ba0dd0de-32a0-4a53-a8ab-3c8804f7af66/limitador/0.log"
Nov 03 13:19:39.563305 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:19:39.563270    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:19:40.296555 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:40.296529    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-csc6h_e135ffb3-aaf9-493b-b0ba-87d09909709e/server/0.log"
Nov 03 13:19:40.528083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:40.528054    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cbwwg_d66fb109-6507-43fe-8916-82bcc32b4ecc/manager/0.log"
Nov 03 13:19:44.006380 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.006353    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:44.086665 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086634    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-credential-socket\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.086834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086721    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-envoy\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.086834 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086791    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-socket\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.086963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086831    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-podinfo\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.086963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086867    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istiod-ca-cert\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.086963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086921    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-credential-socket" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:44.086963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086941    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-certs\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.087185 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.086976    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-token\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.087185 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087026    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-data\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.087185 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087069    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-kube-api-access-npnbw\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-kube-api-access-npnbw\") pod \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\" (UID: \"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b\") "
Nov 03 13:19:44.087185 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087121    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-socket" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:44.087185 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087136    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-certs" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:44.087442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087322    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-data" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:44.087442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087319    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istiod-ca-cert" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:44.087442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087388    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.087442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087415    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.087442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.087433    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.089125 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.089094    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-token" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:44.089230 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.089167    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-kube-api-access-npnbw" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "kube-api-access-npnbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:44.089355 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.089337    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-envoy" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:44.089414 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.089350    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-podinfo" pod "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" (UID: "b7eb47df-a81c-4419-8787-a3dfc6b7dd8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:19:44.188457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.188399    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.188457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.188421    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.188457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.188432    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.188457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.188441    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.188457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.188451    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.188457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.188460    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npnbw\" (UniqueName: \"kubernetes.io/projected/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b-kube-api-access-npnbw\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:44.207401 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.207374    4909 generic.go:356] "Generic (PLEG): container finished" podID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" containerID="8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2" exitCode=0
Nov 03 13:19:44.207522 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.207420    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"
Nov 03 13:19:44.207522 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.207460    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" event={"ID":"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b","Type":"ContainerDied","Data":"8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2"}
Nov 03 13:19:44.207522 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.207512    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh" event={"ID":"b7eb47df-a81c-4419-8787-a3dfc6b7dd8b","Type":"ContainerDied","Data":"15f578bf3d8fb085a5d82e684970797be8f00c1e9117b4b6561da14475bb7f47"}
Nov 03 13:19:44.207683 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.207534    4909 scope.go:122] "RemoveContainer" containerID="8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2"
Nov 03 13:19:44.216453 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.216433    4909 scope.go:122] "RemoveContainer" containerID="8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2"
Nov 03 13:19:44.216754 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:19:44.216732    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2\": container with ID starting with 8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2 not found: ID does not exist" containerID="8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2"
Nov 03 13:19:44.216845 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.216759    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2"} err="failed to get container status \"8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2\": rpc error: code = NotFound desc = could not find container \"8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2\": container with ID starting with 8fcc8b362d9284c3ec65a1bae559d639f156a39d2a0a604e8e0d7fb6c3c008c2 not found: ID does not exist"
Nov 03 13:19:44.358574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.358544    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"]
Nov 03 13:19:44.362995 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:44.362973    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-846mh"]
Nov 03 13:19:45.122634 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:45.122590    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" path="/var/lib/kubelet/pods/b7eb47df-a81c-4419-8787-a3dfc6b7dd8b/volumes"
Nov 03 13:19:53.298417 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.298385    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"]
Nov 03 13:19:53.298976 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.298954    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" containerName="istio-proxy"
Nov 03 13:19:53.299059 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.298977    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" containerName="istio-proxy"
Nov 03 13:19:53.299118 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.299104    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7eb47df-a81c-4419-8787-a3dfc6b7dd8b" containerName="istio-proxy"
Nov 03 13:19:53.303758 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.303739    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.365828 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.365801    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxsb\" (UniqueName: \"kubernetes.io/projected/293ec046-6c1a-4f0d-bf04-1f631f284329-kube-api-access-lhxsb\") pod \"payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") " pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.365949 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.365834    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293ec046-6c1a-4f0d-bf04-1f631f284329-plugins-config-volume\") pod \"payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") " pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.400106 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.400080    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67188486\""
Nov 03 13:19:53.400225 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.400154    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67188486-dockercfg-tpqzn\""
Nov 03 13:19:53.401912 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.401887    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"]
Nov 03 13:19:53.467246 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.467221    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxsb\" (UniqueName: \"kubernetes.io/projected/293ec046-6c1a-4f0d-bf04-1f631f284329-kube-api-access-lhxsb\") pod \"payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") " pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.467363 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.467251    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293ec046-6c1a-4f0d-bf04-1f631f284329-plugins-config-volume\") pod \"payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") " pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.467936 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.467918    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293ec046-6c1a-4f0d-bf04-1f631f284329-plugins-config-volume\") pod \"payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") " pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.521420 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.521368    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxsb\" (UniqueName: \"kubernetes.io/projected/293ec046-6c1a-4f0d-bf04-1f631f284329-kube-api-access-lhxsb\") pod \"payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") " pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.616502 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.616471    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:53.976599 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:53.976572    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"]
Nov 03 13:19:53.979127 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:19:53.979095    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293ec046_6c1a_4f0d_bf04_1f631f284329.slice/crio-0ce4fababec32013ca63b6171ba34c1e59ad5fe5fba9ec034e5bac9b6a64fece WatchSource:0}: Error finding container 0ce4fababec32013ca63b6171ba34c1e59ad5fe5fba9ec034e5bac9b6a64fece: Status 404 returned error can't find the container with id 0ce4fababec32013ca63b6171ba34c1e59ad5fe5fba9ec034e5bac9b6a64fece
Nov 03 13:19:54.252823 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:54.252730    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg" event={"ID":"293ec046-6c1a-4f0d-bf04-1f631f284329","Type":"ContainerStarted","Data":"383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d"}
Nov 03 13:19:54.252823 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:54.252783    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg" event={"ID":"293ec046-6c1a-4f0d-bf04-1f631f284329","Type":"ContainerStarted","Data":"0ce4fababec32013ca63b6171ba34c1e59ad5fe5fba9ec034e5bac9b6a64fece"}
Nov 03 13:19:54.252823 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:54.252805    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:19:54.358548 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:54.358494    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg" podStartSLOduration=1.358477065 podStartE2EDuration="1.358477065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:54.357427874 +0000 UTC m=+5343.369894548" watchObservedRunningTime="2026-11-03 13:19:54.358477065 +0000 UTC m=+5343.370943728"
Nov 03 13:19:55.168245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.168206    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"]
Nov 03 13:19:55.171780 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.171757    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.220005 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.219982    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-67188486-serving-cert\""
Nov 03 13:19:55.267666 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.267634    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"]
Nov 03 13:19:55.282543 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.282513    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgvs\" (UniqueName: \"kubernetes.io/projected/940490c9-703b-46ca-8617-b4c4dd034c17-kube-api-access-dzgvs\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.282658 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.282557    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940490c9-703b-46ca-8617-b4c4dd034c17-openshift-service-ca\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.282751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.282669    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-metrics-certs\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.282751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.282726    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.384391 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.384330    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgvs\" (UniqueName: \"kubernetes.io/projected/940490c9-703b-46ca-8617-b4c4dd034c17-kube-api-access-dzgvs\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.384957 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.384933    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940490c9-703b-46ca-8617-b4c4dd034c17-openshift-service-ca\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.385203 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.385185    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-metrics-certs\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.385323 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.385309    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.385654 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:19:55.385635    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-67188486-serving-cert: secret "maas-api-e2e-ait-67188486-serving-cert" not found
Nov 03 13:19:55.385843 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:19:55.385831    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls podName:940490c9-703b-46ca-8617-b4c4dd034c17 nodeName:}" failed. No retries permitted until 2026-11-03 13:19:55.885809883 +0000 UTC m=+5344.898276542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls") pod "maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh" (UID: "940490c9-703b-46ca-8617-b4c4dd034c17") : secret "maas-api-e2e-ait-67188486-serving-cert" not found
Nov 03 13:19:55.387111 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.387088    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940490c9-703b-46ca-8617-b4c4dd034c17-openshift-service-ca\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.391598 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.390670    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-metrics-certs\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.439959 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.439932    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgvs\" (UniqueName: \"kubernetes.io/projected/940490c9-703b-46ca-8617-b4c4dd034c17-kube-api-access-dzgvs\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.889278 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.889188    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:55.891676 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:55.891647    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls\") pod \"maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:56.084657 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:56.084625    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:56.446351 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:56.446326    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"]
Nov 03 13:19:56.448420 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:19:56.448380    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940490c9_703b_46ca_8617_b4c4dd034c17.slice/crio-83039682816cabaf70133427e44a368a5e4c0c0ffdd7ffa7866b6c1f18d4a949 WatchSource:0}: Error finding container 83039682816cabaf70133427e44a368a5e4c0c0ffdd7ffa7866b6c1f18d4a949: Status 404 returned error can't find the container with id 83039682816cabaf70133427e44a368a5e4c0c0ffdd7ffa7866b6c1f18d4a949
Nov 03 13:19:56.449909 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:56.449890    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:19:57.267167 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:57.267134    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh" event={"ID":"940490c9-703b-46ca-8617-b4c4dd034c17","Type":"ContainerStarted","Data":"83039682816cabaf70133427e44a368a5e4c0c0ffdd7ffa7866b6c1f18d4a949"}
Nov 03 13:19:58.273445 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:58.273406    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh" event={"ID":"940490c9-703b-46ca-8617-b4c4dd034c17","Type":"ContainerStarted","Data":"f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81"}
Nov 03 13:19:58.273843 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:58.273537    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:19:58.380566 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:19:58.380516    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh" podStartSLOduration=2.430364519 podStartE2EDuration="3.380501946s" totalImagesPullingTime="950.137427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:58.380156385 +0000 UTC m=+5347.392623059" watchObservedRunningTime="2026-11-03 13:19:58.380501946 +0000 UTC m=+5347.392968608"
Nov 03 13:20:03.300786 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:03.300682    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:20:04.285575 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:04.285544    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:20:05.260348 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:05.260318    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:20:09.562631 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:20:09.562595    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:20:10.047404 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:10.047355    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"]
Nov 03 13:20:10.093154 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:10.093125    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedeacaf-rhqqq"]
Nov 03 13:20:11.121548 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:11.121518    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b74642-9783-4509-8eaa-0b2c33f0f8e2" path="/var/lib/kubelet/pods/b4b74642-9783-4509-8eaa-0b2c33f0f8e2/volumes"
Nov 03 13:20:21.043052 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:21.043019    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"]
Nov 03 13:20:21.089162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:21.089128    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be06a04-6gdzr"]
Nov 03 13:20:21.121452 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:21.121418    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3638d1-389f-41b8-8fa1-0d3053a39f63" path="/var/lib/kubelet/pods/ef3638d1-389f-41b8-8fa1-0d3053a39f63/volumes"
Nov 03 13:20:37.523714 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:37.523675    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"]
Nov 03 13:20:39.562902 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:20:39.562868    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:20:40.008553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.008523    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"]
Nov 03 13:20:40.008765 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.008718    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh" podUID="940490c9-703b-46ca-8617-b4c4dd034c17" containerName="maas-api" containerID="cri-o://f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81" gracePeriod=30
Nov 03 13:20:40.258238 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.258211    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:20:40.372531 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.372437    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls\") pod \"940490c9-703b-46ca-8617-b4c4dd034c17\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") "
Nov 03 13:20:40.372737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.372556    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-metrics-certs\") pod \"940490c9-703b-46ca-8617-b4c4dd034c17\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") "
Nov 03 13:20:40.372737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.372638    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940490c9-703b-46ca-8617-b4c4dd034c17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940490c9-703b-46ca-8617-b4c4dd034c17-openshift-service-ca\") pod \"940490c9-703b-46ca-8617-b4c4dd034c17\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") "
Nov 03 13:20:40.372737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.372682    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940490c9-703b-46ca-8617-b4c4dd034c17-kube-api-access-dzgvs\" (UniqueName: \"kubernetes.io/projected/940490c9-703b-46ca-8617-b4c4dd034c17-kube-api-access-dzgvs\") pod \"940490c9-703b-46ca-8617-b4c4dd034c17\" (UID: \"940490c9-703b-46ca-8617-b4c4dd034c17\") "
Nov 03 13:20:40.373325 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.373291    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940490c9-703b-46ca-8617-b4c4dd034c17-openshift-service-ca" pod "940490c9-703b-46ca-8617-b4c4dd034c17" (UID: "940490c9-703b-46ca-8617-b4c4dd034c17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:40.375491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.375467    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls" pod "940490c9-703b-46ca-8617-b4c4dd034c17" (UID: "940490c9-703b-46ca-8617-b4c4dd034c17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:40.375608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.375578    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-metrics-certs" pod "940490c9-703b-46ca-8617-b4c4dd034c17" (UID: "940490c9-703b-46ca-8617-b4c4dd034c17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:40.375675 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.375622    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940490c9-703b-46ca-8617-b4c4dd034c17-kube-api-access-dzgvs" pod "940490c9-703b-46ca-8617-b4c4dd034c17" (UID: "940490c9-703b-46ca-8617-b4c4dd034c17"). InnerVolumeSpecName "kube-api-access-dzgvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:40.462570 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.462544    4909 generic.go:356] "Generic (PLEG): container finished" podID="940490c9-703b-46ca-8617-b4c4dd034c17" containerID="f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81" exitCode=0
Nov 03 13:20:40.462683 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.462586    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"
Nov 03 13:20:40.462683 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.462588    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh" event={"ID":"940490c9-703b-46ca-8617-b4c4dd034c17","Type":"ContainerDied","Data":"f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81"}
Nov 03 13:20:40.462796 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.462690    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh" event={"ID":"940490c9-703b-46ca-8617-b4c4dd034c17","Type":"ContainerDied","Data":"83039682816cabaf70133427e44a368a5e4c0c0ffdd7ffa7866b6c1f18d4a949"}
Nov 03 13:20:40.462796 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.462743    4909 scope.go:122] "RemoveContainer" containerID="f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81"
Nov 03 13:20:40.471746 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.471724    4909 scope.go:122] "RemoveContainer" containerID="f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81"
Nov 03 13:20:40.471999 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:20:40.471981    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81\": container with ID starting with f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81 not found: ID does not exist" containerID="f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81"
Nov 03 13:20:40.472054 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.472009    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81"} err="failed to get container status \"f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81\": rpc error: code = NotFound desc = could not find container \"f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81\": container with ID starting with f56e6d524e6a214ea368a2a4665ca823ec9424d6b9e84e1c0ea3b64ec6e7ff81 not found: ID does not exist"
Nov 03 13:20:40.473604 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.473581    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:40.473656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.473612    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940490c9-703b-46ca-8617-b4c4dd034c17-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:40.473656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.473628    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgvs\" (UniqueName: \"kubernetes.io/projected/940490c9-703b-46ca-8617-b4c4dd034c17-kube-api-access-dzgvs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:40.473656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.473643    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940490c9-703b-46ca-8617-b4c4dd034c17-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:40.613673 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.613639    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"]
Nov 03 13:20:40.616832 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:40.616806    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67188486-86b6bc5d7-6p4mh"]
Nov 03 13:20:41.121518 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:41.121489    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940490c9-703b-46ca-8617-b4c4dd034c17" path="/var/lib/kubelet/pods/940490c9-703b-46ca-8617-b4c4dd034c17/volumes"
Nov 03 13:20:42.524024 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.523989    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg" podUID="293ec046-6c1a-4f0d-bf04-1f631f284329" containerName="payload-processing" containerID="cri-o://383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d" gracePeriod=25
Nov 03 13:20:42.774544 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.774490    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:20:42.893234 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.893208    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293ec046-6c1a-4f0d-bf04-1f631f284329-kube-api-access-lhxsb\" (UniqueName: \"kubernetes.io/projected/293ec046-6c1a-4f0d-bf04-1f631f284329-kube-api-access-lhxsb\") pod \"293ec046-6c1a-4f0d-bf04-1f631f284329\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") "
Nov 03 13:20:42.893387 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.893258    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293ec046-6c1a-4f0d-bf04-1f631f284329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293ec046-6c1a-4f0d-bf04-1f631f284329-plugins-config-volume\") pod \"293ec046-6c1a-4f0d-bf04-1f631f284329\" (UID: \"293ec046-6c1a-4f0d-bf04-1f631f284329\") "
Nov 03 13:20:42.893634 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.893611    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293ec046-6c1a-4f0d-bf04-1f631f284329-plugins-config-volume" pod "293ec046-6c1a-4f0d-bf04-1f631f284329" (UID: "293ec046-6c1a-4f0d-bf04-1f631f284329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:42.895368 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.895337    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293ec046-6c1a-4f0d-bf04-1f631f284329-kube-api-access-lhxsb" pod "293ec046-6c1a-4f0d-bf04-1f631f284329" (UID: "293ec046-6c1a-4f0d-bf04-1f631f284329"). InnerVolumeSpecName "kube-api-access-lhxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:42.993904 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.993869    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxsb\" (UniqueName: \"kubernetes.io/projected/293ec046-6c1a-4f0d-bf04-1f631f284329-kube-api-access-lhxsb\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:42.993904 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:42.993904    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293ec046-6c1a-4f0d-bf04-1f631f284329-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:43.477959 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.477927    4909 generic.go:356] "Generic (PLEG): container finished" podID="293ec046-6c1a-4f0d-bf04-1f631f284329" containerID="383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d" exitCode=0
Nov 03 13:20:43.478129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.477973    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"
Nov 03 13:20:43.478129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.477993    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg" event={"ID":"293ec046-6c1a-4f0d-bf04-1f631f284329","Type":"ContainerDied","Data":"383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d"}
Nov 03 13:20:43.478129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.478052    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg" event={"ID":"293ec046-6c1a-4f0d-bf04-1f631f284329","Type":"ContainerDied","Data":"0ce4fababec32013ca63b6171ba34c1e59ad5fe5fba9ec034e5bac9b6a64fece"}
Nov 03 13:20:43.478129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.478075    4909 scope.go:122] "RemoveContainer" containerID="383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d"
Nov 03 13:20:43.486593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.486575    4909 scope.go:122] "RemoveContainer" containerID="383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d"
Nov 03 13:20:43.486873 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:20:43.486855    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d\": container with ID starting with 383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d not found: ID does not exist" containerID="383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d"
Nov 03 13:20:43.486936 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.486883    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d"} err="failed to get container status \"383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d\": rpc error: code = NotFound desc = could not find container \"383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d\": container with ID starting with 383ab9bfa6b4ef6603854f2a5f133788f106e94d6d4446ecf7b1a83978121f7d not found: ID does not exist"
Nov 03 13:20:43.629538 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.629506    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"]
Nov 03 13:20:43.675205 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:43.675177    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67188486-78ff7f5c99-ksdfg"]
Nov 03 13:20:44.603120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:44.603084    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:20:45.121240 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:45.121207    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293ec046-6c1a-4f0d-bf04-1f631f284329" path="/var/lib/kubelet/pods/293ec046-6c1a-4f0d-bf04-1f631f284329/volumes"
Nov 03 13:20:51.831481 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:51.831455    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:20:51.843222 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:20:51.843201    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:21:09.565685 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:21:09.565644    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:21:17.019913 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.019880    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"]
Nov 03 13:21:17.020393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.020261    4909 cpu_manager.go:414] "containerMap: removing container" podUID="940490c9-703b-46ca-8617-b4c4dd034c17" containerName="maas-api"
Nov 03 13:21:17.020393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.020274    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940490c9-703b-46ca-8617-b4c4dd034c17" containerName="maas-api"
Nov 03 13:21:17.020393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.020314    4909 cpu_manager.go:414] "containerMap: removing container" podUID="293ec046-6c1a-4f0d-bf04-1f631f284329" containerName="payload-processing"
Nov 03 13:21:17.020393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.020319    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293ec046-6c1a-4f0d-bf04-1f631f284329" containerName="payload-processing"
Nov 03 13:21:17.020393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.020394    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="293ec046-6c1a-4f0d-bf04-1f631f284329" containerName="payload-processing"
Nov 03 13:21:17.020590 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.020404    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="940490c9-703b-46ca-8617-b4c4dd034c17" containerName="maas-api"
Nov 03 13:21:17.024630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.024610    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089544 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089514    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988fz\" (UniqueName: \"kubernetes.io/projected/925da08a-493d-409c-8476-7721001ae76f-kube-api-access-988fz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089551    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089568    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089686 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089674    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089690    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925da08a-493d-409c-8476-7721001ae76f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089730    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925da08a-493d-409c-8476-7721001ae76f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089745    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089837 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089796    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925da08a-493d-409c-8476-7721001ae76f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.089991 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.089909    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.114762 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.114733    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"]
Nov 03 13:21:17.191180 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191152    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925da08a-493d-409c-8476-7721001ae76f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191180 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191183    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191441 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191281    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925da08a-493d-409c-8476-7721001ae76f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191441 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191367    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191441 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191405    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988fz\" (UniqueName: \"kubernetes.io/projected/925da08a-493d-409c-8476-7721001ae76f-kube-api-access-988fz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191441 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191436    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191693 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191462    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191693 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191527    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191693 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191551    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925da08a-493d-409c-8476-7721001ae76f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191908 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191813    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.191908 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.191827    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.192087 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.192062    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.192166 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.192115    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.192166 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.192132    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925da08a-493d-409c-8476-7721001ae76f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.193491 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.193468    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925da08a-493d-409c-8476-7721001ae76f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.193809 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.193790    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925da08a-493d-409c-8476-7721001ae76f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.245178 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.245152    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925da08a-493d-409c-8476-7721001ae76f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.245707 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.245680    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-988fz\" (UniqueName: \"kubernetes.io/projected/925da08a-493d-409c-8476-7721001ae76f-kube-api-access-988fz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bwfnv\" (UID: \"925da08a-493d-409c-8476-7721001ae76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.336887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.336832    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:17.744482 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.744442    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"]
Nov 03 13:21:17.748598 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:21:17.748568    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925da08a_493d_409c_8476_7721001ae76f.slice/crio-81fccff08a7bbee6928ab6f83a4258c8ae1834d9fc3915cd6ba357efdec42d5c WatchSource:0}: Error finding container 81fccff08a7bbee6928ab6f83a4258c8ae1834d9fc3915cd6ba357efdec42d5c: Status 404 returned error can't find the container with id 81fccff08a7bbee6928ab6f83a4258c8ae1834d9fc3915cd6ba357efdec42d5c
Nov 03 13:21:17.750676 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.750645    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:21:17.750771 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.750733    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:21:17.750771 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:17.750766    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:21:18.629608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:18.629566    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" event={"ID":"925da08a-493d-409c-8476-7721001ae76f","Type":"ContainerStarted","Data":"34d05f47786507fa68e2cfbc2733fd4bd005412643d9e4a139174fa645edcf6f"}
Nov 03 13:21:18.630036 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:18.629626    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" event={"ID":"925da08a-493d-409c-8476-7721001ae76f","Type":"ContainerStarted","Data":"81fccff08a7bbee6928ab6f83a4258c8ae1834d9fc3915cd6ba357efdec42d5c"}
Nov 03 13:21:18.784250 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:18.784199    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podStartSLOduration=2.784181614 podStartE2EDuration="2.784181614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:18.782374824 +0000 UTC m=+5427.794841486" watchObservedRunningTime="2026-11-03 13:21:18.784181614 +0000 UTC m=+5427.796648279"
Nov 03 13:21:19.337668 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:19.337630    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:19.361720 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:19.357831    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 13:21:19.361720 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:19.357924    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 13:21:20.337279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:20.337244    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 13:21:20.337800 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:20.337304    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 13:21:21.361721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:21.337982    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 13:21:21.361721 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:21.338040    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 13:21:22.337597 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:22.337561    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 13:21:22.337860 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:22.337613    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 13:21:23.338081 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:23.338040    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 13:21:23.338754 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:23.338719    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 13:21:24.337924 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:24.337889    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 13:21:24.338107 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:24.337944    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 13:21:26.338290 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:26.338246    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:21:26.338760 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:26.338341    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:27.380830 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:27.380794    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:21:27.381371 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:27.380846    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:28.187900 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:28.187868    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:28.188145 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:28.188124    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:28.188865 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:28.188845    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv"
Nov 03 13:21:39.563904 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:21:39.563826    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:21:53.293613 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:53.293583    4909 scope.go:122] "RemoveContainer" containerID="0f7b72568caf1cfbfea0b472bffe922c17f9151ebbf75549a5c3970ac2a4c1c1"
Nov 03 13:21:53.301881 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:21:53.301845    4909 scope.go:122] "RemoveContainer" containerID="23ee7c0eba8ee84d2d02ec192f62eac1d36d7e4488e0c535de31758ed1fb0c6d"
Nov 03 13:22:01.339064 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:01.339027    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:22:09.563557 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:22:09.563522    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:22:17.464778 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.464736    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"]
Nov 03 13:22:17.468419 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.468397    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.510497 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.510472    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"]
Nov 03 13:22:17.513755 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.513739    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.532708 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.532677    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.532810 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.532730    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48mh\" (UniqueName: \"kubernetes.io/projected/98367206-ed8d-4a86-b2e0-5850d8e470e1-kube-api-access-r48mh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.532865 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.532830    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.532928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.532862    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.532928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.532888    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.533045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.532950    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.533045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.533014    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.533135 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.533083    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98367206-ed8d-4a86-b2e0-5850d8e470e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.533135 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.533104    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634009 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.633979    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634019    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98367206-ed8d-4a86-b2e0-5850d8e470e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634040    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634143    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634180    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634462 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634211    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634462 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634230    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634462 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634250    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48mh\" (UniqueName: \"kubernetes.io/projected/98367206-ed8d-4a86-b2e0-5850d8e470e1-kube-api-access-r48mh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634462 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634265    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634466    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634505    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634540    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634565    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634595    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.634723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634658    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.634723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634713    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.635124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634758    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdmc\" (UniqueName: \"kubernetes.io/projected/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-kube-api-access-gtdmc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.635124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634789    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.635124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634845    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.635124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634863    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98367206-ed8d-4a86-b2e0-5850d8e470e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.635124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.634948    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.635124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.635032    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.635384 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.635199    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.636589 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.636567    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.636734 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.636716    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.643298 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.643274    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"]
Nov 03 13:22:17.659990 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.659964    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"]
Nov 03 13:22:17.690300 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.690272    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98367206-ed8d-4a86-b2e0-5850d8e470e1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.690466 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.690444    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48mh\" (UniqueName: \"kubernetes.io/projected/98367206-ed8d-4a86-b2e0-5850d8e470e1-kube-api-access-r48mh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vr8qh\" (UID: \"98367206-ed8d-4a86-b2e0-5850d8e470e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.736133 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736081    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736133 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736121    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736282 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736159    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736282 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736187    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdmc\" (UniqueName: \"kubernetes.io/projected/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-kube-api-access-gtdmc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736282 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736228    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736282 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736270    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736301    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736345    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736500 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736389    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736514    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736528    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736573    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.736670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.736652    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.737177 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.737157    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.738442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.738422    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.738779 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.738762    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.781046 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.781025    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:17.788045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.788023    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.788465 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.788446    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdmc\" (UniqueName: \"kubernetes.io/projected/858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d-kube-api-access-gtdmc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f2bh6\" (UID: \"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:17.825738 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:17.825693    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:18.192662 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.192629    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"]
Nov 03 13:22:18.229473 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:22:18.229439    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98367206_ed8d_4a86_b2e0_5850d8e470e1.slice/crio-f6538e9d2a3302a3f44eeae1b3dc67ec8ef2abc20bb6f978250e680f5eb64e6b WatchSource:0}: Error finding container f6538e9d2a3302a3f44eeae1b3dc67ec8ef2abc20bb6f978250e680f5eb64e6b: Status 404 returned error can't find the container with id f6538e9d2a3302a3f44eeae1b3dc67ec8ef2abc20bb6f978250e680f5eb64e6b
Nov 03 13:22:18.231635 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.231605    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:22:18.231718 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.231667    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:22:18.231718 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.231710    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:22:18.242659 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.242634    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"]
Nov 03 13:22:18.246601 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:22:18.246575    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858aa31d_4b48_4cc0_a6d7_e4e9ba1ddd5d.slice/crio-591f26b7778da8d4416433877d91cf9e746d667ab58f8d12f2ff94fa03086803 WatchSource:0}: Error finding container 591f26b7778da8d4416433877d91cf9e746d667ab58f8d12f2ff94fa03086803: Status 404 returned error can't find the container with id 591f26b7778da8d4416433877d91cf9e746d667ab58f8d12f2ff94fa03086803
Nov 03 13:22:18.248739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.248518    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:22:18.248739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.248588    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:22:18.248739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.248623    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:22:18.929744 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.929688    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" event={"ID":"98367206-ed8d-4a86-b2e0-5850d8e470e1","Type":"ContainerStarted","Data":"fee1d04d67749d348f13e254095e0069d811160c6f46a3b3ca812c063caa6fba"}
Nov 03 13:22:18.930234 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.929810    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" event={"ID":"98367206-ed8d-4a86-b2e0-5850d8e470e1","Type":"ContainerStarted","Data":"f6538e9d2a3302a3f44eeae1b3dc67ec8ef2abc20bb6f978250e680f5eb64e6b"}
Nov 03 13:22:18.931857 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.931405    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" event={"ID":"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d","Type":"ContainerStarted","Data":"c06772eb8e858c1b910de546be95ded6dbdacb58a7041acbc5e0a4130a6680b8"}
Nov 03 13:22:18.931857 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:18.931449    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" event={"ID":"858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d","Type":"ContainerStarted","Data":"591f26b7778da8d4416433877d91cf9e746d667ab58f8d12f2ff94fa03086803"}
Nov 03 13:22:19.049521 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:19.049460    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podStartSLOduration=2.049442198 podStartE2EDuration="2.049442198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:19.047602299 +0000 UTC m=+5488.060068962" watchObservedRunningTime="2026-11-03 13:22:19.049442198 +0000 UTC m=+5488.061908869"
Nov 03 13:22:19.785297 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:19.785263    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:19.785647 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:19.785626    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 13:22:19.785819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:19.785796    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 13:22:19.826187 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:19.826166    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:19.827146 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:19.827128    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:22:19.827279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:19.827263    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Nov 03 13:22:20.789400 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:20.789364    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 13:22:20.789947 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:20.789517    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 13:22:20.826959 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:20.826927    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:22:20.827120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:20.826974    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Nov 03 13:22:21.408619 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:21.408556    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podStartSLOduration=4.408540794 podStartE2EDuration="4.408540794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:19.162138474 +0000 UTC m=+5488.174605137" watchObservedRunningTime="2026-11-03 13:22:21.408540794 +0000 UTC m=+5490.421007457"
Nov 03 13:22:21.409926 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:21.409906    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:22:21.782723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:21.782041    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 13:22:21.782723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:21.782102    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 13:22:21.826573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:21.826540    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:22:21.826975 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:21.826585    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Nov 03 13:22:22.787322 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:22.787283    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 13:22:22.787649 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:22.787617    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 13:22:22.826139 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:22.826108    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:22:22.826258 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:22.826153    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Nov 03 13:22:23.786363 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:23.786328    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 13:22:23.791553 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:23.791028    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 13:22:23.826753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:23.826727    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:22:23.826879 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:23.826772    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Nov 03 13:22:24.781782 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:24.781745    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 13:22:24.781962 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:24.781806    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 13:22:24.826232 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:24.826200    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:22:24.826569 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:24.826274    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Nov 03 13:22:26.782959 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:26.782913    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:26.783425 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:26.783003    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:26.826812 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:26.826777    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:26.826950 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:26.826853    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:27.827859 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:27.827823    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:27.828360 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:27.828014    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:27.874719 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:27.873861    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:27.874719 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:27.873950    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:28.871857 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:28.871806    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:28.872498 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:28.872451    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:28.918398 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:28.917939    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:28.918398 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:28.918000    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:29.881723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:29.880784    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:29.881723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:29.880864    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:29.919304 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:29.919265    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:29.919599 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:29.919562    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:30.922844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:30.922806    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:30.923330 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:30.922860    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:30.961884 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:30.961851    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:30.962134 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:30.962098    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:31.923398 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:31.923362    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:31.923905 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:31.923454    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:32.008095 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.007997    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:32.008276 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.008153    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:32.047140 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.047110    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:32.047552 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.047520    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:32.078409 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.078379    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6"
Nov 03 13:22:32.173524 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.173448    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:32.173783 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.173762    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:32.174408 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.174391    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh"
Nov 03 13:22:32.218751 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:32.218727    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:22:39.691725 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:22:39.679980    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:22:41.803917 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:41.803884    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:22:50.513307 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:22:50.513272    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:23:09.562893 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:23:09.562811    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:23:27.630518 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:23:27.630486    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:23:29.239723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:23:29.238798    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:23:29.239723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:23:29.238872    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:39.563083 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:23:39.563044    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:23:41.488877 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:23:41.488846    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:24:09.564071 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:24:09.564033    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:24:18.724984 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:24:18.724950    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:24:39.562375 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:24:39.562289    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:24:43.193173 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:24:43.193137    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:24:54.418245 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:24:54.418211    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:25:01.702646 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.702608    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"]
Nov 03 13:25:01.706314 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.706292    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.755814 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.755781    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-55973f-openshift-default-dockercfg-dzdtv\""
Nov 03 13:25:01.818091 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.818060    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"]
Nov 03 13:25:01.824763 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.824732    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-token\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.824900 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.824771    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-data\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.824900 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.824859    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf6edf-e053-4438-916b-87c2fa7ca971-istiod-ca-cert\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.824996 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.824902    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-certs\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.824996 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.824926    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-podinfo\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.825083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.825020    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-envoy\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.825083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.825047    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-socket\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.825083 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.825071    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdslg\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-kube-api-access-jdslg\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.825250 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.825195    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-credential-socket\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.925862 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.925826    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-certs\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926060 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.925874    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-podinfo\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926060 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.925916    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-envoy\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926060 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.925935    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-socket\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926060 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.925961    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdslg\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-kube-api-access-jdslg\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926060 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926003    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-credential-socket\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926060 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926033    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-token\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926065    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-data\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926128    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf6edf-e053-4438-916b-87c2fa7ca971-istiod-ca-cert\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926280    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-certs\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926336    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-socket\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926669 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926530    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-data\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.926669 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.926576    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-credential-socket\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.927068 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.927043    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf6edf-e053-4438-916b-87c2fa7ca971-istiod-ca-cert\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.928310 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.928290    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-envoy\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.928805 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.928784    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-podinfo\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.979756 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.979661    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-token\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:01.979892 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:01.979867    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdslg\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-kube-api-access-jdslg\") pod \"e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") " pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:02.018475 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.018447    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:02.448414 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.448388    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"]
Nov 03 13:25:02.450253 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:25:02.450224    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cf6edf_e053_4438_916b_87c2fa7ca971.slice/crio-2af8fc62f86b650ea21fafff8c09a79b241b6cf42cac928029b099558fc19bc4 WatchSource:0}: Error finding container 2af8fc62f86b650ea21fafff8c09a79b241b6cf42cac928029b099558fc19bc4: Status 404 returned error can't find the container with id 2af8fc62f86b650ea21fafff8c09a79b241b6cf42cac928029b099558fc19bc4
Nov 03 13:25:02.452439 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.452407    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:25:02.452513 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.452499    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:25:02.452559 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.452531    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:25:02.880334 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.880244    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" event={"ID":"b3cf6edf-e053-4438-916b-87c2fa7ca971","Type":"ContainerStarted","Data":"8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f"}
Nov 03 13:25:02.880334 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.880301    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" event={"ID":"b3cf6edf-e053-4438-916b-87c2fa7ca971","Type":"ContainerStarted","Data":"2af8fc62f86b650ea21fafff8c09a79b241b6cf42cac928029b099558fc19bc4"}
Nov 03 13:25:02.988848 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:02.988772    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" podStartSLOduration=1.988759506 podStartE2EDuration="1.988759506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:02.987438072 +0000 UTC m=+5651.999904734" watchObservedRunningTime="2026-11-03 13:25:02.988759506 +0000 UTC m=+5652.001226180"
Nov 03 13:25:03.018594 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:03.018565    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:03.023902 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:03.023873    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:03.884742 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:03.884688    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:03.885608 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:03.885589    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:09.520507 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.520470    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"]
Nov 03 13:25:09.524069 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.524053    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.562962 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:25:09.562930    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:25:09.628150 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.628115    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7137d0c6-dockercfg-94hp6\""
Nov 03 13:25:09.629001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.628967    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-55973f\""
Nov 03 13:25:09.629220 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.629053    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"]
Nov 03 13:25:09.645621 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.645589    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2418d13d-fecf-4a6b-9a63-3d9745205b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.645819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.645746    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmqp7\" (UniqueName: \"kubernetes.io/projected/2418d13d-fecf-4a6b-9a63-3d9745205b1c-kube-api-access-lmqp7\") pod \"payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.685849 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.685810    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"]
Nov 03 13:25:09.689310 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.689292    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:09.744460 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.744431    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"]
Nov 03 13:25:09.747267 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.747242    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmqp7\" (UniqueName: \"kubernetes.io/projected/2418d13d-fecf-4a6b-9a63-3d9745205b1c-kube-api-access-lmqp7\") pod \"payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.747410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.747380    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2418d13d-fecf-4a6b-9a63-3d9745205b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.748104 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.748086    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2418d13d-fecf-4a6b-9a63-3d9745205b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.802908 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.802859    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmqp7\" (UniqueName: \"kubernetes.io/projected/2418d13d-fecf-4a6b-9a63-3d9745205b1c-kube-api-access-lmqp7\") pod \"payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.836838 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.836813    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:09.848896 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.848868    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecaa387-6a0d-4bbe-a2f7-03304734431a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:09.849012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.848949    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z56k\" (UniqueName: \"kubernetes.io/projected/5ecaa387-6a0d-4bbe-a2f7-03304734431a-kube-api-access-2z56k\") pod \"payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:09.950102 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.950065    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecaa387-6a0d-4bbe-a2f7-03304734431a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:09.950262 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.950125    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z56k\" (UniqueName: \"kubernetes.io/projected/5ecaa387-6a0d-4bbe-a2f7-03304734431a-kube-api-access-2z56k\") pod \"payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:09.950794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:09.950776    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecaa387-6a0d-4bbe-a2f7-03304734431a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:10.003626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.003592    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z56k\" (UniqueName: \"kubernetes.io/projected/5ecaa387-6a0d-4bbe-a2f7-03304734431a-kube-api-access-2z56k\") pod \"payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:10.199830 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.199806    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"]
Nov 03 13:25:10.202363 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:25:10.202336    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2418d13d_fecf_4a6b_9a63_3d9745205b1c.slice/crio-ff0f0d58549f131016f68516b5aa54a720161572a4c25d2bcd0fa1fd47be2830 WatchSource:0}: Error finding container ff0f0d58549f131016f68516b5aa54a720161572a4c25d2bcd0fa1fd47be2830: Status 404 returned error can't find the container with id ff0f0d58549f131016f68516b5aa54a720161572a4c25d2bcd0fa1fd47be2830
Nov 03 13:25:10.302101 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.302072    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:10.665163 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.665137    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"]
Nov 03 13:25:10.667442 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:25:10.667413    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecaa387_6a0d_4bbe_a2f7_03304734431a.slice/crio-56dfbe60df8889676a1b5b40dc706f26b6f6f0d1791be4326860a161125ba93c WatchSource:0}: Error finding container 56dfbe60df8889676a1b5b40dc706f26b6f6f0d1791be4326860a161125ba93c: Status 404 returned error can't find the container with id 56dfbe60df8889676a1b5b40dc706f26b6f6f0d1791be4326860a161125ba93c
Nov 03 13:25:10.927174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.927070    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" event={"ID":"2418d13d-fecf-4a6b-9a63-3d9745205b1c","Type":"ContainerStarted","Data":"5d6a6633b92e967505ffaa504967c620cbe80a087fd2bb2c5c286ec65a6f7f71"}
Nov 03 13:25:10.927174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.927136    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" event={"ID":"2418d13d-fecf-4a6b-9a63-3d9745205b1c","Type":"ContainerStarted","Data":"ff0f0d58549f131016f68516b5aa54a720161572a4c25d2bcd0fa1fd47be2830"}
Nov 03 13:25:10.927386 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.927242    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:10.928595 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.928576    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" event={"ID":"5ecaa387-6a0d-4bbe-a2f7-03304734431a","Type":"ContainerStarted","Data":"6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045"}
Nov 03 13:25:10.928730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.928607    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" event={"ID":"5ecaa387-6a0d-4bbe-a2f7-03304734431a","Type":"ContainerStarted","Data":"56dfbe60df8889676a1b5b40dc706f26b6f6f0d1791be4326860a161125ba93c"}
Nov 03 13:25:10.928730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:10.928637    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:11.034178 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:11.034134    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" podStartSLOduration=2.034120755 podStartE2EDuration="2.034120755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:11.033032641 +0000 UTC m=+5660.045499303" watchObservedRunningTime="2026-11-03 13:25:11.034120755 +0000 UTC m=+5660.046587418"
Nov 03 13:25:11.139024 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:11.138920    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" podStartSLOduration=2.138904446 podStartE2EDuration="2.138904446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:11.137744959 +0000 UTC m=+5660.150211623" watchObservedRunningTime="2026-11-03 13:25:11.138904446 +0000 UTC m=+5660.151371109"
Nov 03 13:25:11.994211 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:11.994172    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"]
Nov 03 13:25:11.998007 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:11.997982    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.047508 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.047455    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-55973f-serving-cert\""
Nov 03 13:25:12.087768 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.087743    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72e2787-978f-4586-9b42-443d4a6d804d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.087907 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.087802    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.087963 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.087935    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.088012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.087992    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhm44\" (UniqueName: \"kubernetes.io/projected/f72e2787-978f-4586-9b42-443d4a6d804d-kube-api-access-bhm44\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.095941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.095912    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"]
Nov 03 13:25:12.189490 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.189455    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.189645 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.189501    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhm44\" (UniqueName: \"kubernetes.io/projected/f72e2787-978f-4586-9b42-443d4a6d804d-kube-api-access-bhm44\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.189645 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.189556    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72e2787-978f-4586-9b42-443d4a6d804d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.189645 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.189597    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.189794 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:25:12.189687    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-serving-cert: secret "maas-api-e2e-worker-single-main-55973f-serving-cert" not found
Nov 03 13:25:12.189794 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:25:12.189766    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls podName:f72e2787-978f-4586-9b42-443d4a6d804d nodeName:}" failed. No retries permitted until 2026-11-03 13:25:12.68974824 +0000 UTC m=+5661.702214904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls") pod "maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx" (UID: "f72e2787-978f-4586-9b42-443d4a6d804d") : secret "maas-api-e2e-worker-single-main-55973f-serving-cert" not found
Nov 03 13:25:12.190287 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.190265    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72e2787-978f-4586-9b42-443d4a6d804d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.192008 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.191982    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.243383 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.243347    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhm44\" (UniqueName: \"kubernetes.io/projected/f72e2787-978f-4586-9b42-443d4a6d804d-kube-api-access-bhm44\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.694206 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.694171    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.696978 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.696948    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:12.916856 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:12.916817    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:13.280994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:13.280917    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"]
Nov 03 13:25:13.286011 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:13.285990    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:25:13.943542 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:13.943507    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx" event={"ID":"f72e2787-978f-4586-9b42-443d4a6d804d","Type":"ContainerStarted","Data":"11ed1105915555fee92d968a5f72a02e32bc83e20d757393121bd0dcdaf7aa65"}
Nov 03 13:25:14.954674 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:14.954635    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx" event={"ID":"f72e2787-978f-4586-9b42-443d4a6d804d","Type":"ContainerStarted","Data":"aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631"}
Nov 03 13:25:14.955122 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:14.954709    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:15.063444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:15.063394    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx" podStartSLOduration=3.068486824 podStartE2EDuration="4.063379732s" totalImagesPullingTime="994.892908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:15.062017546 +0000 UTC m=+5664.074484222" watchObservedRunningTime="2026-11-03 13:25:15.063379732 +0000 UTC m=+5664.075846396"
Nov 03 13:25:20.094341 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:20.094304    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:25:20.965792 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:20.965762    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:25:21.936021 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:21.935992    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:25:21.936419 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:21.936061    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:25:30.032155 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:30.032123    4909 generic.go:356] "Generic (PLEG): container finished" podID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerID="8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f" exitCode=0
Nov 03 13:25:30.032600 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:30.032195    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" event={"ID":"b3cf6edf-e053-4438-916b-87c2fa7ca971","Type":"ContainerDied","Data":"8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f"}
Nov 03 13:25:30.032600 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:30.032566    4909 scope.go:122] "RemoveContainer" containerID="8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f"
Nov 03 13:25:30.033898 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:30.033868    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:25:30.033962 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:30.033948    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:25:30.033998 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:30.033988    4909 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 13:25:31.063278 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:31.063235    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" event={"ID":"b3cf6edf-e053-4438-916b-87c2fa7ca971","Type":"ContainerStarted","Data":"c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb"}
Nov 03 13:25:32.018609 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:32.018573    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:32.019624 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:32.019598    4909 patch_prober.go:29] interesting pod/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Nov 03 13:25:32.019758 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:32.019652    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Nov 03 13:25:33.018964 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:33.018926    4909 patch_prober.go:29] interesting pod/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Nov 03 13:25:33.019371 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:33.018983    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Nov 03 13:25:33.885442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:33.885408    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:34.019002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:34.018969    4909 patch_prober.go:29] interesting pod/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Nov 03 13:25:34.019497 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:34.019025    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Nov 03 13:25:35.019367 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:35.019320    4909 patch_prober.go:29] interesting pod/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Nov 03 13:25:35.019819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:35.019403    4909 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Nov 03 13:25:36.060134 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:36.060094    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:36.063234 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:36.063208    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:25:39.336230 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:39.336196    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:25:39.566589 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:25:39.566550    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:25:51.861759 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:51.861733    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:25:51.874523 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:25:51.874500    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:26:06.496908 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:06.496825    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:26:09.563158 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:26:09.563123    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:26:32.338889 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:32.338856    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:26:36.066537 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:36.066507    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"]
Nov 03 13:26:36.353564 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:36.353491    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"]
Nov 03 13:26:38.505742 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:38.505710    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"]
Nov 03 13:26:38.506274 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:38.505898    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx" podUID="f72e2787-978f-4586-9b42-443d4a6d804d" containerName="maas-api" containerID="cri-o://aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631" gracePeriod=30
Nov 03 13:26:39.589627 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:26:39.589587    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:26:39.758044 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.758017    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:26:39.848958 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.848894    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-metrics-certs\") pod \"f72e2787-978f-4586-9b42-443d4a6d804d\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") "
Nov 03 13:26:39.848958 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.848934    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72e2787-978f-4586-9b42-443d4a6d804d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72e2787-978f-4586-9b42-443d4a6d804d-openshift-service-ca\") pod \"f72e2787-978f-4586-9b42-443d4a6d804d\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") "
Nov 03 13:26:39.849120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.849061    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72e2787-978f-4586-9b42-443d4a6d804d-kube-api-access-bhm44\" (UniqueName: \"kubernetes.io/projected/f72e2787-978f-4586-9b42-443d4a6d804d-kube-api-access-bhm44\") pod \"f72e2787-978f-4586-9b42-443d4a6d804d\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") "
Nov 03 13:26:39.849120 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.849102    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls\") pod \"f72e2787-978f-4586-9b42-443d4a6d804d\" (UID: \"f72e2787-978f-4586-9b42-443d4a6d804d\") "
Nov 03 13:26:39.849310 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.849287    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72e2787-978f-4586-9b42-443d4a6d804d-openshift-service-ca" pod "f72e2787-978f-4586-9b42-443d4a6d804d" (UID: "f72e2787-978f-4586-9b42-443d4a6d804d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:39.849476 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.849459    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72e2787-978f-4586-9b42-443d4a6d804d-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:39.851010 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.850985    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-metrics-certs" pod "f72e2787-978f-4586-9b42-443d4a6d804d" (UID: "f72e2787-978f-4586-9b42-443d4a6d804d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:39.851132 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.851088    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls" pod "f72e2787-978f-4586-9b42-443d4a6d804d" (UID: "f72e2787-978f-4586-9b42-443d4a6d804d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:39.851200 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.851154    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72e2787-978f-4586-9b42-443d4a6d804d-kube-api-access-bhm44" pod "f72e2787-978f-4586-9b42-443d4a6d804d" (UID: "f72e2787-978f-4586-9b42-443d4a6d804d"). InnerVolumeSpecName "kube-api-access-bhm44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:39.950149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.950117    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:39.950149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.950148    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhm44\" (UniqueName: \"kubernetes.io/projected/f72e2787-978f-4586-9b42-443d4a6d804d-kube-api-access-bhm44\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:39.950342 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:39.950158    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72e2787-978f-4586-9b42-443d4a6d804d-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:40.396462 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.396428    4909 generic.go:356] "Generic (PLEG): container finished" podID="f72e2787-978f-4586-9b42-443d4a6d804d" containerID="aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631" exitCode=0
Nov 03 13:26:40.396641 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.396484    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"
Nov 03 13:26:40.396641 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.396512    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx" event={"ID":"f72e2787-978f-4586-9b42-443d4a6d804d","Type":"ContainerDied","Data":"aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631"}
Nov 03 13:26:40.396641 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.396584    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx" event={"ID":"f72e2787-978f-4586-9b42-443d4a6d804d","Type":"ContainerDied","Data":"11ed1105915555fee92d968a5f72a02e32bc83e20d757393121bd0dcdaf7aa65"}
Nov 03 13:26:40.396641 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.396606    4909 scope.go:122] "RemoveContainer" containerID="aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631"
Nov 03 13:26:40.405838 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.405813    4909 scope.go:122] "RemoveContainer" containerID="aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631"
Nov 03 13:26:40.406101 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:26:40.406083    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631\": container with ID starting with aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631 not found: ID does not exist" containerID="aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631"
Nov 03 13:26:40.406149 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.406110    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631"} err="failed to get container status \"aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631\": rpc error: code = NotFound desc = could not find container \"aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631\": container with ID starting with aa874354e672b225782c04c2886bc08d3936a524e4893ad3780afdfada05f631 not found: ID does not exist"
Nov 03 13:26:40.547482 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.547458    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"]
Nov 03 13:26:40.593560 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:40.593532    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-55973f-5d54f86975-gnnvx"]
Nov 03 13:26:41.067445 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:41.067387    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" podUID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" containerName="payload-processing" containerID="cri-o://6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045" gracePeriod=25
Nov 03 13:26:41.122133 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:41.122100    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72e2787-978f-4586-9b42-443d4a6d804d" path="/var/lib/kubelet/pods/f72e2787-978f-4586-9b42-443d4a6d804d/volumes"
Nov 03 13:26:41.354390 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:41.354300    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" podUID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" containerName="payload-pre-processing" containerID="cri-o://5d6a6633b92e967505ffaa504967c620cbe80a087fd2bb2c5c286ec65a6f7f71" gracePeriod=25
Nov 03 13:26:41.934807 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:41.934763    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" podUID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:26:41.935224 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:41.934764    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" podUID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:26:42.439430 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:42.439400    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:26:51.934937 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:51.934902    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" podUID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:26:51.935325 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:26:51.934904    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" podUID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:27:01.323815 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.323753    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:27:01.343112 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.343085    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ecaa387-6a0d-4bbe-a2f7-03304734431a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecaa387-6a0d-4bbe-a2f7-03304734431a-plugins-config-volume\") pod \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") "
Nov 03 13:27:01.343240 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.343143    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecaa387-6a0d-4bbe-a2f7-03304734431a-kube-api-access-2z56k\" (UniqueName: \"kubernetes.io/projected/5ecaa387-6a0d-4bbe-a2f7-03304734431a-kube-api-access-2z56k\") pod \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\" (UID: \"5ecaa387-6a0d-4bbe-a2f7-03304734431a\") "
Nov 03 13:27:01.343437 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.343415    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecaa387-6a0d-4bbe-a2f7-03304734431a-plugins-config-volume" pod "5ecaa387-6a0d-4bbe-a2f7-03304734431a" (UID: "5ecaa387-6a0d-4bbe-a2f7-03304734431a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:01.345256 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.345229    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecaa387-6a0d-4bbe-a2f7-03304734431a-kube-api-access-2z56k" pod "5ecaa387-6a0d-4bbe-a2f7-03304734431a" (UID: "5ecaa387-6a0d-4bbe-a2f7-03304734431a"). InnerVolumeSpecName "kube-api-access-2z56k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:01.444137 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.444107    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecaa387-6a0d-4bbe-a2f7-03304734431a-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:01.444137 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.444135    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z56k\" (UniqueName: \"kubernetes.io/projected/5ecaa387-6a0d-4bbe-a2f7-03304734431a-kube-api-access-2z56k\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:01.507113 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.507079    4909 generic.go:356] "Generic (PLEG): container finished" podID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" containerID="5d6a6633b92e967505ffaa504967c620cbe80a087fd2bb2c5c286ec65a6f7f71" exitCode=0
Nov 03 13:27:01.507302 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.507165    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" event={"ID":"2418d13d-fecf-4a6b-9a63-3d9745205b1c","Type":"ContainerDied","Data":"5d6a6633b92e967505ffaa504967c620cbe80a087fd2bb2c5c286ec65a6f7f71"}
Nov 03 13:27:01.508734 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.508685    4909 generic.go:356] "Generic (PLEG): container finished" podID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" containerID="6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045" exitCode=0
Nov 03 13:27:01.508850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.508731    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" event={"ID":"5ecaa387-6a0d-4bbe-a2f7-03304734431a","Type":"ContainerDied","Data":"6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045"}
Nov 03 13:27:01.508850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.508765    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"
Nov 03 13:27:01.508850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.508789    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d" event={"ID":"5ecaa387-6a0d-4bbe-a2f7-03304734431a","Type":"ContainerDied","Data":"56dfbe60df8889676a1b5b40dc706f26b6f6f0d1791be4326860a161125ba93c"}
Nov 03 13:27:01.508850 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.508821    4909 scope.go:122] "RemoveContainer" containerID="6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045"
Nov 03 13:27:01.517933 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.517917    4909 scope.go:122] "RemoveContainer" containerID="6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045"
Nov 03 13:27:01.518229 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:27:01.518211    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045\": container with ID starting with 6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045 not found: ID does not exist" containerID="6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045"
Nov 03 13:27:01.518300 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.518242    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045"} err="failed to get container status \"6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045\": rpc error: code = NotFound desc = could not find container \"6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045\": container with ID starting with 6c6f651ca4a2a5c3bc6720536cec45a1a28bd9662d04b35ddf22f9eb07a12045 not found: ID does not exist"
Nov 03 13:27:01.596196 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.596171    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:27:01.646757 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.646721    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2418d13d-fecf-4a6b-9a63-3d9745205b1c-kube-api-access-lmqp7\" (UniqueName: \"kubernetes.io/projected/2418d13d-fecf-4a6b-9a63-3d9745205b1c-kube-api-access-lmqp7\") pod \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") "
Nov 03 13:27:01.646911 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.646782    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2418d13d-fecf-4a6b-9a63-3d9745205b1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2418d13d-fecf-4a6b-9a63-3d9745205b1c-plugins-config-volume\") pod \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\" (UID: \"2418d13d-fecf-4a6b-9a63-3d9745205b1c\") "
Nov 03 13:27:01.647162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.647134    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2418d13d-fecf-4a6b-9a63-3d9745205b1c-plugins-config-volume" pod "2418d13d-fecf-4a6b-9a63-3d9745205b1c" (UID: "2418d13d-fecf-4a6b-9a63-3d9745205b1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:01.648867 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.648847    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2418d13d-fecf-4a6b-9a63-3d9745205b1c-kube-api-access-lmqp7" pod "2418d13d-fecf-4a6b-9a63-3d9745205b1c" (UID: "2418d13d-fecf-4a6b-9a63-3d9745205b1c"). InnerVolumeSpecName "kube-api-access-lmqp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:01.660667 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.660645    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"]
Nov 03 13:27:01.664688 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.664659    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-55973f-86bcc45869pd2d"]
Nov 03 13:27:01.748049 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.748021    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmqp7\" (UniqueName: \"kubernetes.io/projected/2418d13d-fecf-4a6b-9a63-3d9745205b1c-kube-api-access-lmqp7\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:01.748049 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:01.748047    4909 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2418d13d-fecf-4a6b-9a63-3d9745205b1c-plugins-config-volume\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:02.514985 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:02.514955    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"
Nov 03 13:27:02.514985 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:02.514976    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6" event={"ID":"2418d13d-fecf-4a6b-9a63-3d9745205b1c","Type":"ContainerDied","Data":"ff0f0d58549f131016f68516b5aa54a720161572a4c25d2bcd0fa1fd47be2830"}
Nov 03 13:27:02.515518 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:02.515043    4909 scope.go:122] "RemoveContainer" containerID="5d6a6633b92e967505ffaa504967c620cbe80a087fd2bb2c5c286ec65a6f7f71"
Nov 03 13:27:02.665887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:02.665856    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"]
Nov 03 13:27:02.711980 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:02.711870    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-55973f-7448cxj9z6"]
Nov 03 13:27:03.122885 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:03.122851    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" path="/var/lib/kubelet/pods/2418d13d-fecf-4a6b-9a63-3d9745205b1c/volumes"
Nov 03 13:27:03.123246 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:03.123233    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" path="/var/lib/kubelet/pods/5ecaa387-6a0d-4bbe-a2f7-03304734431a/volumes"
Nov 03 13:27:09.563864 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:27:09.563830    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:27:16.737032 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:16.736998    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"]
Nov 03 13:27:16.737448 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:16.737210    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy" containerID="cri-o://c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb" gracePeriod=30
Nov 03 13:27:21.063022 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:21.062985    4909 patch_prober.go:29] interesting pod/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:27:21.063410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:21.063060    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:27:21.992722 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:21.992672    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:27:22.126561 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126481    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-token\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126583    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-data\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126630    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-podinfo\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126682    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-credential-socket\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126769    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-certs\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126812    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cf6edf-e053-4438-916b-87c2fa7ca971-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf6edf-e053-4438-916b-87c2fa7ca971-istiod-ca-cert\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126857    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-envoy\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126891    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-kube-api-access-jdslg\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-kube-api-access-jdslg\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.126916    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-data" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:22.127042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127022    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-socket\") pod \"b3cf6edf-e053-4438-916b-87c2fa7ca971\" (UID: \"b3cf6edf-e053-4438-916b-87c2fa7ca971\") "
Nov 03 13:27:22.127514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127046    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-credential-socket" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:22.127514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127087    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-certs" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:22.127514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127365    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-socket" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:22.127514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127377    4909 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-credential-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.127514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127400    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cf6edf-e053-4438-916b-87c2fa7ca971-istiod-ca-cert" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:22.127514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127413    4909 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.127514 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.127431    4909 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-data\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.128977 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.128948    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-envoy" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:22.129108 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.129075    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-kube-api-access-jdslg" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "kube-api-access-jdslg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:22.129207 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.129188    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-token" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:22.129264 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.129246    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-podinfo" pod "b3cf6edf-e053-4438-916b-87c2fa7ca971" (UID: "b3cf6edf-e053-4438-916b-87c2fa7ca971"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:27:22.227941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.227911    4909 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-podinfo\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.227941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.227935    4909 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf6edf-e053-4438-916b-87c2fa7ca971-istiod-ca-cert\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.227941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.227946    4909 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-envoy\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.228150 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.227956    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdslg\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-kube-api-access-jdslg\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.228150 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.227967    4909 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf6edf-e053-4438-916b-87c2fa7ca971-workload-socket\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.228150 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.227975    4909 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf6edf-e053-4438-916b-87c2fa7ca971-istio-token\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:22.607924 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.607894    4909 generic.go:356] "Generic (PLEG): container finished" podID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerID="c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb" exitCode=0
Nov 03 13:27:22.608124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.607960    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"
Nov 03 13:27:22.608124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.607991    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" event={"ID":"b3cf6edf-e053-4438-916b-87c2fa7ca971","Type":"ContainerDied","Data":"c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb"}
Nov 03 13:27:22.608124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.608051    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5" event={"ID":"b3cf6edf-e053-4438-916b-87c2fa7ca971","Type":"ContainerDied","Data":"2af8fc62f86b650ea21fafff8c09a79b241b6cf42cac928029b099558fc19bc4"}
Nov 03 13:27:22.608124 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.608083    4909 scope.go:122] "RemoveContainer" containerID="c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb"
Nov 03 13:27:22.618129 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.618109    4909 scope.go:122] "RemoveContainer" containerID="8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f"
Nov 03 13:27:22.628416 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.628397    4909 scope.go:122] "RemoveContainer" containerID="c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb"
Nov 03 13:27:22.628739 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:27:22.628714    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb\": container with ID starting with c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb not found: ID does not exist" containerID="c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb"
Nov 03 13:27:22.628799 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.628749    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb"} err="failed to get container status \"c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb\": rpc error: code = NotFound desc = could not find container \"c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb\": container with ID starting with c80bcf66effcc505b14468425eacaa8a8c4823237415079bcba9cbf53876a8bb not found: ID does not exist"
Nov 03 13:27:22.628799 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.628770    4909 scope.go:122] "RemoveContainer" containerID="8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f"
Nov 03 13:27:22.629068 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:27:22.629047    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f\": container with ID starting with 8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f not found: ID does not exist" containerID="8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f"
Nov 03 13:27:22.629128 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.629075    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f"} err="failed to get container status \"8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f\": rpc error: code = NotFound desc = could not find container \"8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f\": container with ID starting with 8ffc8dfaa646233f273e22c891b3c21a1f16c7af1d35203b732f86930fc6805f not found: ID does not exist"
Nov 03 13:27:22.764040 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.764006    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"]
Nov 03 13:27:22.767827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:22.767805    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-55973f-openshift-default-5f5fc69797dhmq5"]
Nov 03 13:27:23.129066 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:23.123743    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" path="/var/lib/kubelet/pods/b3cf6edf-e053-4438-916b-87c2fa7ca971/volumes"
Nov 03 13:27:36.596951 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:36.596865    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:27:39.637815 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:27:39.637776    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:27:55.575515 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575478    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"]
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575861    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575872    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575887    4909 cpu_manager.go:414] "containerMap: removing container" podUID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" containerName="payload-pre-processing"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575893    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" containerName="payload-pre-processing"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575902    4909 cpu_manager.go:414] "containerMap: removing container" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575906    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575920    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" containerName="payload-processing"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575924    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" containerName="payload-processing"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575941    4909 cpu_manager.go:414] "containerMap: removing container" podUID="f72e2787-978f-4586-9b42-443d4a6d804d" containerName="maas-api"
Nov 03 13:27:55.575944 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.575946    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72e2787-978f-4586-9b42-443d4a6d804d" containerName="maas-api"
Nov 03 13:27:55.576280 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.576002    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="2418d13d-fecf-4a6b-9a63-3d9745205b1c" containerName="payload-pre-processing"
Nov 03 13:27:55.576280 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.576011    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy"
Nov 03 13:27:55.576280 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.576018    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecaa387-6a0d-4bbe-a2f7-03304734431a" containerName="payload-processing"
Nov 03 13:27:55.576280 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.576025    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72e2787-978f-4586-9b42-443d4a6d804d" containerName="maas-api"
Nov 03 13:27:55.580356 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.580329    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.669370 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.669332    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-model-cache\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.669370 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.669372    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-home\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.669574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.669441    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-tmp-dir\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.669574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.669504    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-dshm\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.669656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.669594    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfqx\" (UniqueName: \"kubernetes.io/projected/7230edbf-2528-4325-aa45-15d50fc73621-kube-api-access-9cfqx\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.669656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.669617    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7230edbf-2528-4325-aa45-15d50fc73621-tls-certs\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.676535 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.676518    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9aad7fec-kserve-self-signed-certs\""
Nov 03 13:27:55.678121 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.678101    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"]
Nov 03 13:27:55.770339 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770300    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-model-cache\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770546 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770347    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-home\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770546 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770380    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-tmp-dir\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770546 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770440    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-dshm\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770546 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770508    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfqx\" (UniqueName: \"kubernetes.io/projected/7230edbf-2528-4325-aa45-15d50fc73621-kube-api-access-9cfqx\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770546 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770529    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7230edbf-2528-4325-aa45-15d50fc73621-tls-certs\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770865 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770774    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-model-cache\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770865 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770800    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-home\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.770988 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.770864    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-tmp-dir\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.772750 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.772732    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-dshm\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.773174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.773153    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7230edbf-2528-4325-aa45-15d50fc73621-tls-certs\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.825281 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.825248    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfqx\" (UniqueName: \"kubernetes.io/projected/7230edbf-2528-4325-aa45-15d50fc73621-kube-api-access-9cfqx\") pod \"e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") " pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:55.900547 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:55.900488    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:56.290538 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:56.290515    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"]
Nov 03 13:27:56.292795 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:27:56.292766    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7230edbf_2528_4325_aa45_15d50fc73621.slice/crio-2934a595d49bb0febd0ab2d24a276be06283e1fc0930413eb6a6422af5762ff4 WatchSource:0}: Error finding container 2934a595d49bb0febd0ab2d24a276be06283e1fc0930413eb6a6422af5762ff4: Status 404 returned error can't find the container with id 2934a595d49bb0febd0ab2d24a276be06283e1fc0930413eb6a6422af5762ff4
Nov 03 13:27:56.796757 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:56.796720    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx" event={"ID":"7230edbf-2528-4325-aa45-15d50fc73621","Type":"ContainerStarted","Data":"b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565"}
Nov 03 13:27:56.797227 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:56.796763    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx" event={"ID":"7230edbf-2528-4325-aa45-15d50fc73621","Type":"ContainerStarted","Data":"2934a595d49bb0febd0ab2d24a276be06283e1fc0930413eb6a6422af5762ff4"}
Nov 03 13:27:56.900831 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:56.900804    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:27:56.906827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:27:56.906787    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx" podStartSLOduration=1.906774576 podStartE2EDuration="1.906774576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:56.904356084 +0000 UTC m=+5825.916822747" watchObservedRunningTime="2026-11-03 13:27:56.906774576 +0000 UTC m=+5825.919241238"
Nov 03 13:28:00.230437 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:00.230405    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:28:03.135540 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:03.134863    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:03.135540 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:03.134945    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:28:03.218941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:03.218793    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:03.218941 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:03.218867    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:28:05.901525 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:05.901487    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:28:05.914366 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:05.914340    4909 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:28:06.874931 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:06.874897    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:28:09.563467 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:28:09.563426    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:28:26.514219 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:26.514182    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:28:29.250723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:29.246793    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:29.250723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:29.246896    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:28:39.564008 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:28:39.563970    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:28:41.900135 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:41.900101    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:28:47.739210 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:47.739173    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"]
Nov 03 13:28:57.122343 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:28:57.122311    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:29:02.759753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:02.759650    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx" podUID="7230edbf-2528-4325-aa45-15d50fc73621" containerName="main" containerID="cri-o://b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565" gracePeriod=45
Nov 03 13:29:02.864476 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:02.864440    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx" podUID="7230edbf-2528-4325-aa45-15d50fc73621" containerName="main" probeResult="failure" output="Get \"https://10.132.0.234:8000/ready\": dial tcp 10.132.0.234:8000: connect: connection refused"
Nov 03 13:29:03.018565 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.018497    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:29:03.110162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110129    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7230edbf-2528-4325-aa45-15d50fc73621-kube-api-access-9cfqx\" (UniqueName: \"kubernetes.io/projected/7230edbf-2528-4325-aa45-15d50fc73621-kube-api-access-9cfqx\") pod \"7230edbf-2528-4325-aa45-15d50fc73621\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") "
Nov 03 13:29:03.110330 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110229    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-dshm\") pod \"7230edbf-2528-4325-aa45-15d50fc73621\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") "
Nov 03 13:29:03.110330 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110289    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-tmp-dir\") pod \"7230edbf-2528-4325-aa45-15d50fc73621\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") "
Nov 03 13:29:03.110330 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110310    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7230edbf-2528-4325-aa45-15d50fc73621-tls-certs\" (UniqueName: \"kubernetes.io/secret/7230edbf-2528-4325-aa45-15d50fc73621-tls-certs\") pod \"7230edbf-2528-4325-aa45-15d50fc73621\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") "
Nov 03 13:29:03.110330 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110329    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-home\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-home\") pod \"7230edbf-2528-4325-aa45-15d50fc73621\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") "
Nov 03 13:29:03.110547 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110350    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-model-cache\") pod \"7230edbf-2528-4325-aa45-15d50fc73621\" (UID: \"7230edbf-2528-4325-aa45-15d50fc73621\") "
Nov 03 13:29:03.110547 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110514    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-tmp-dir" pod "7230edbf-2528-4325-aa45-15d50fc73621" (UID: "7230edbf-2528-4325-aa45-15d50fc73621"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:03.110693 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110663    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-home" pod "7230edbf-2528-4325-aa45-15d50fc73621" (UID: "7230edbf-2528-4325-aa45-15d50fc73621"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:03.110791 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.110671    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-model-cache" pod "7230edbf-2528-4325-aa45-15d50fc73621" (UID: "7230edbf-2528-4325-aa45-15d50fc73621"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:03.112513 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.112489    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7230edbf-2528-4325-aa45-15d50fc73621-tls-certs" pod "7230edbf-2528-4325-aa45-15d50fc73621" (UID: "7230edbf-2528-4325-aa45-15d50fc73621"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:29:03.112611 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.112494    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-dshm" pod "7230edbf-2528-4325-aa45-15d50fc73621" (UID: "7230edbf-2528-4325-aa45-15d50fc73621"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:03.112656 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.112612    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7230edbf-2528-4325-aa45-15d50fc73621-kube-api-access-9cfqx" pod "7230edbf-2528-4325-aa45-15d50fc73621" (UID: "7230edbf-2528-4325-aa45-15d50fc73621"). InnerVolumeSpecName "kube-api-access-9cfqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:03.211431 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.211398    4909 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-tmp-dir\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:03.211431 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.211428    4909 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7230edbf-2528-4325-aa45-15d50fc73621-tls-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:03.211431 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.211437    4909 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-home\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:03.211671 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.211446    4909 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-model-cache\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:03.211671 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.211455    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cfqx\" (UniqueName: \"kubernetes.io/projected/7230edbf-2528-4325-aa45-15d50fc73621-kube-api-access-9cfqx\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:03.211671 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.211465    4909 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7230edbf-2528-4325-aa45-15d50fc73621-dshm\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:03.239793 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.239765    4909 generic.go:356] "Generic (PLEG): container finished" podID="7230edbf-2528-4325-aa45-15d50fc73621" containerID="b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565" exitCode=0
Nov 03 13:29:03.239943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.239810    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx" event={"ID":"7230edbf-2528-4325-aa45-15d50fc73621","Type":"ContainerDied","Data":"b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565"}
Nov 03 13:29:03.239943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.239817    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"
Nov 03 13:29:03.239943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.239843    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx" event={"ID":"7230edbf-2528-4325-aa45-15d50fc73621","Type":"ContainerDied","Data":"2934a595d49bb0febd0ab2d24a276be06283e1fc0930413eb6a6422af5762ff4"}
Nov 03 13:29:03.239943 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.239863    4909 scope.go:122] "RemoveContainer" containerID="b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565"
Nov 03 13:29:03.248426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.248407    4909 scope.go:122] "RemoveContainer" containerID="b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565"
Nov 03 13:29:03.248811 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:29:03.248789    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565\": container with ID starting with b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565 not found: ID does not exist" containerID="b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565"
Nov 03 13:29:03.248878 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.248822    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565"} err="failed to get container status \"b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565\": rpc error: code = NotFound desc = could not find container \"b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565\": container with ID starting with b05ee0aa71bf0fccecbd4705984700360aaa11e5c62e4bdab348ef447c7d9565 not found: ID does not exist"
Nov 03 13:29:03.391119 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.391054    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"]
Nov 03 13:29:03.394116 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:03.394092    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9aad7fec-kserve-f96c9bd9-wfrmx"]
Nov 03 13:29:05.122150 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:05.122117    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7230edbf-2528-4325-aa45-15d50fc73621" path="/var/lib/kubelet/pods/7230edbf-2528-4325-aa45-15d50fc73621/volumes"
Nov 03 13:29:06.397279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:06.397242    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:29:09.562823 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:29:09.562784    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:29:16.914893 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:16.914860    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:29:31.392509 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:29:31.392475    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:29:39.563537 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:29:39.563495    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:30:04.258178 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.258143    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f7b9fc877-jqd6f"]
Nov 03 13:30:04.258652 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.258559    4909 cpu_manager.go:414] "containerMap: removing container" podUID="7230edbf-2528-4325-aa45-15d50fc73621" containerName="main"
Nov 03 13:30:04.258652 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.258571    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7230edbf-2528-4325-aa45-15d50fc73621" containerName="main"
Nov 03 13:30:04.258772 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.258654    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cf6edf-e053-4438-916b-87c2fa7ca971" containerName="istio-proxy"
Nov 03 13:30:04.258772 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.258664    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="7230edbf-2528-4325-aa45-15d50fc73621" containerName="main"
Nov 03 13:30:04.262984 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.262965    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.313179 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.313158    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 13:30:04.313179 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.313171    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xs659\""
Nov 03 13:30:04.316234 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.316211    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f7b9fc877-jqd6f"]
Nov 03 13:30:04.385368 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.385343    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbdbdac-bc9c-4945-8375-4aaac24d79e0-plugins-config-volume\") pod \"payload-processing-5f7b9fc877-jqd6f\" (UID: \"6cbdbdac-bc9c-4945-8375-4aaac24d79e0\") " pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.385508 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.385408    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzkm\" (UniqueName: \"kubernetes.io/projected/6cbdbdac-bc9c-4945-8375-4aaac24d79e0-kube-api-access-bgzkm\") pod \"payload-processing-5f7b9fc877-jqd6f\" (UID: \"6cbdbdac-bc9c-4945-8375-4aaac24d79e0\") " pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.486634 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.486606    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzkm\" (UniqueName: \"kubernetes.io/projected/6cbdbdac-bc9c-4945-8375-4aaac24d79e0-kube-api-access-bgzkm\") pod \"payload-processing-5f7b9fc877-jqd6f\" (UID: \"6cbdbdac-bc9c-4945-8375-4aaac24d79e0\") " pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.486778 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.486667    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbdbdac-bc9c-4945-8375-4aaac24d79e0-plugins-config-volume\") pod \"payload-processing-5f7b9fc877-jqd6f\" (UID: \"6cbdbdac-bc9c-4945-8375-4aaac24d79e0\") " pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.487328 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.487308    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbdbdac-bc9c-4945-8375-4aaac24d79e0-plugins-config-volume\") pod \"payload-processing-5f7b9fc877-jqd6f\" (UID: \"6cbdbdac-bc9c-4945-8375-4aaac24d79e0\") " pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.539496 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.539433    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzkm\" (UniqueName: \"kubernetes.io/projected/6cbdbdac-bc9c-4945-8375-4aaac24d79e0-kube-api-access-bgzkm\") pod \"payload-processing-5f7b9fc877-jqd6f\" (UID: \"6cbdbdac-bc9c-4945-8375-4aaac24d79e0\") " pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.574818 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.574789    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:04.938478 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:04.938451    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f7b9fc877-jqd6f"]
Nov 03 13:30:04.940630 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:30:04.940600    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbdbdac_bc9c_4945_8375_4aaac24d79e0.slice/crio-e01586cebd9771a0bbf883362b2995a71998e58fccbbae573dd2a7a52ae69fe9 WatchSource:0}: Error finding container e01586cebd9771a0bbf883362b2995a71998e58fccbbae573dd2a7a52ae69fe9: Status 404 returned error can't find the container with id e01586cebd9771a0bbf883362b2995a71998e58fccbbae573dd2a7a52ae69fe9
Nov 03 13:30:05.556044 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:05.555984    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f" event={"ID":"6cbdbdac-bc9c-4945-8375-4aaac24d79e0","Type":"ContainerStarted","Data":"bda3d6112c4efab35fca4b73803119fe6de761baef3b9f15513b884be8d68247"}
Nov 03 13:30:05.556509 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:05.556056    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f" event={"ID":"6cbdbdac-bc9c-4945-8375-4aaac24d79e0","Type":"ContainerStarted","Data":"e01586cebd9771a0bbf883362b2995a71998e58fccbbae573dd2a7a52ae69fe9"}
Nov 03 13:30:05.556509 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:05.556124    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:05.663123 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:05.663067    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f" podStartSLOduration=1.6630487600000001 podStartE2EDuration="1.66304876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:05.660354623 +0000 UTC m=+5954.672821297" watchObservedRunningTime="2026-11-03 13:30:05.66304876 +0000 UTC m=+5954.675515424"
Nov 03 13:30:09.563343 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:30:09.563303    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:30:16.563070 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:16.563037    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f7b9fc877-jqd6f"
Nov 03 13:30:24.513256 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:24.513221    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:30:39.565768 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:30:39.565677    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:30:51.888544 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:51.888518    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:30:51.902289 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:30:51.902268    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:31:09.563581 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:31:09.563544    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:31:39.563555 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:31:39.563515    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:32:09.563854 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:32:09.563817    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:32:11.092141 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:32:11.092111    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:32:21.608730 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:32:21.608665    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:32:23.997228 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:32:23.997189    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:32:23.997860 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:32:23.997825    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:32:39.564213 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:32:39.564177    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:32:39.792366 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:32:39.792335    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:32:50.507002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:32:50.506964    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:33:08.515995 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:33:08.515962    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:33:09.564474 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:33:09.564441    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:33:21.596294 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:33:21.596256    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:33:39.563951 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:33:39.563868    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:33:51.909827 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:33:51.909793    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:34:02.501057 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:34:02.501023    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:34:09.577083 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:34:09.577038    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:34:14.923408 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:34:14.923368    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:34:37.129351 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:34:37.129319    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:34:39.564443 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:34:39.564407    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:34:46.725444 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:34:46.725405    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:34:55.330052 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:34:55.330016    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:35:05.391794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:05.391712    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:35:09.564258 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:35:09.564222    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:35:14.825584 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:14.825548    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:35:23.933652 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:23.933613    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:35:33.395239 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:33.395204    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:35:39.563674 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:35:39.563635    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:35:50.628041 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:50.628008    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:35:51.927839 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:51.927802    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:35:51.942724 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:51.941421    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:35:59.239727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:59.238798    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:35:59.239727 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:35:59.238881    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:36:01.426654 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:36:01.426618    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:36:09.563542 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:36:09.563506    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:36:14.605424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:36:14.605391    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:36:24.845542 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:36:24.845505    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:36:36.932012 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:36:36.931924    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:36:39.563431 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:36:39.563392    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:36:57.999634 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:36:57.999594    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:37:08.629267 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:08.629233    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:37:09.564266 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:37:09.564229    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:37:32.738067 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:32.738033    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:37:39.563934 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:37:39.563893    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:37:44.609993 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.609955    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:37:44.812071 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.812039    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"]
Nov 03 13:37:44.815544 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.815522    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:44.866620 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.866563    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a6706a-serving-cert\""
Nov 03 13:37:44.912997 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.912973    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"]
Nov 03 13:37:44.918601 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.918579    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baee21a-8b1f-4942-9593-5104f2981e18-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:44.918661 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.918624    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:44.918661 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.918651    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:44.918825 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:44.918801    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7w5\" (UniqueName: \"kubernetes.io/projected/5baee21a-8b1f-4942-9593-5104f2981e18-kube-api-access-4s7w5\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.019742 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.019691    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.019896 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.019760    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.019896 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.019786    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7w5\" (UniqueName: \"kubernetes.io/projected/5baee21a-8b1f-4942-9593-5104f2981e18-kube-api-access-4s7w5\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.019896 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.019848    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baee21a-8b1f-4942-9593-5104f2981e18-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.020031 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:37:45.019892    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-serving-cert: secret "maas-api-e2e-worker-single-main-a6706a-serving-cert" not found
Nov 03 13:37:45.020031 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:37:45.019968    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls podName:5baee21a-8b1f-4942-9593-5104f2981e18 nodeName:}" failed. No retries permitted until 2026-11-03 13:37:45.519944674 +0000 UTC m=+6414.532411319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls") pod "maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" (UID: "5baee21a-8b1f-4942-9593-5104f2981e18") : secret "maas-api-e2e-worker-single-main-a6706a-serving-cert" not found
Nov 03 13:37:45.020472 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.020455    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baee21a-8b1f-4942-9593-5104f2981e18-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.022161 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.022136    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.081506 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.081470    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7w5\" (UniqueName: \"kubernetes.io/projected/5baee21a-8b1f-4942-9593-5104f2981e18-kube-api-access-4s7w5\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.527550 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.527513    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.529928 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.529908    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:45.728490 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:45.728462    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:46.094544 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:46.094516    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"]
Nov 03 13:37:46.097345 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:37:46.097306    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baee21a_8b1f_4942_9593_5104f2981e18.slice/crio-c4442d68a51a74c56c492e9e509f5e17494c4c1870459dd6e23bba9225300ed6 WatchSource:0}: Error finding container c4442d68a51a74c56c492e9e509f5e17494c4c1870459dd6e23bba9225300ed6: Status 404 returned error can't find the container with id c4442d68a51a74c56c492e9e509f5e17494c4c1870459dd6e23bba9225300ed6
Nov 03 13:37:46.098638 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:46.098620    4909 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:37:46.135657 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:46.135627    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" event={"ID":"5baee21a-8b1f-4942-9593-5104f2981e18","Type":"ContainerStarted","Data":"c4442d68a51a74c56c492e9e509f5e17494c4c1870459dd6e23bba9225300ed6"}
Nov 03 13:37:47.142252 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:47.142216    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" event={"ID":"5baee21a-8b1f-4942-9593-5104f2981e18","Type":"ContainerStarted","Data":"40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58"}
Nov 03 13:37:47.142626 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:47.142278    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:47.300305 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:47.300254    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" podStartSLOduration=2.321873374 podStartE2EDuration="3.300240395s" totalImagesPullingTime="978.367021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:47.299559485 +0000 UTC m=+6416.312026149" watchObservedRunningTime="2026-11-03 13:37:47.300240395 +0000 UTC m=+6416.312707054"
Nov 03 13:37:53.153164 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:53.153136    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:37:56.858866 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:37:56.858828    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:38:09.563360 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:38:09.563309    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:38:11.519353 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:38:11.519311    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:38:39.563430 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:38:39.563393    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:38:39.590937 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:38:39.590909    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:39:08.724561 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:08.724527    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:39:09.562995 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:39:09.562961    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:39:12.837865 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:12.837832    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"]
Nov 03 13:39:12.838414 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:12.838025    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" podUID="5baee21a-8b1f-4942-9593-5104f2981e18" containerName="maas-api" containerID="cri-o://40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58" gracePeriod=30
Nov 03 13:39:13.148757 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:13.148663    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" podUID="5baee21a-8b1f-4942-9593-5104f2981e18" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.236:8443/health\": dial tcp 10.132.0.236:8443: connect: connection refused"
Nov 03 13:39:14.098021 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.097993    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:39:14.257380 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.257352    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls\") pod \"5baee21a-8b1f-4942-9593-5104f2981e18\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") "
Nov 03 13:39:14.257538 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.257421    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5baee21a-8b1f-4942-9593-5104f2981e18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baee21a-8b1f-4942-9593-5104f2981e18-openshift-service-ca\") pod \"5baee21a-8b1f-4942-9593-5104f2981e18\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") "
Nov 03 13:39:14.257538 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.257495    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5baee21a-8b1f-4942-9593-5104f2981e18-kube-api-access-4s7w5\" (UniqueName: \"kubernetes.io/projected/5baee21a-8b1f-4942-9593-5104f2981e18-kube-api-access-4s7w5\") pod \"5baee21a-8b1f-4942-9593-5104f2981e18\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") "
Nov 03 13:39:14.257630 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.257547    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-metrics-certs\") pod \"5baee21a-8b1f-4942-9593-5104f2981e18\" (UID: \"5baee21a-8b1f-4942-9593-5104f2981e18\") "
Nov 03 13:39:14.257886 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.257855    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baee21a-8b1f-4942-9593-5104f2981e18-openshift-service-ca" pod "5baee21a-8b1f-4942-9593-5104f2981e18" (UID: "5baee21a-8b1f-4942-9593-5104f2981e18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:14.259729 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.259679    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-metrics-certs" pod "5baee21a-8b1f-4942-9593-5104f2981e18" (UID: "5baee21a-8b1f-4942-9593-5104f2981e18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:14.259836 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.259797    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls" pod "5baee21a-8b1f-4942-9593-5104f2981e18" (UID: "5baee21a-8b1f-4942-9593-5104f2981e18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:14.259836 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.259806    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baee21a-8b1f-4942-9593-5104f2981e18-kube-api-access-4s7w5" pod "5baee21a-8b1f-4942-9593-5104f2981e18" (UID: "5baee21a-8b1f-4942-9593-5104f2981e18"). InnerVolumeSpecName "kube-api-access-4s7w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:14.358573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.358541    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baee21a-8b1f-4942-9593-5104f2981e18-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:14.358573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.358568    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s7w5\" (UniqueName: \"kubernetes.io/projected/5baee21a-8b1f-4942-9593-5104f2981e18-kube-api-access-4s7w5\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:14.358777 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.358580    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:14.358777 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.358589    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baee21a-8b1f-4942-9593-5104f2981e18-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:14.545574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.545501    4909 generic.go:356] "Generic (PLEG): container finished" podID="5baee21a-8b1f-4942-9593-5104f2981e18" containerID="40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58" exitCode=0
Nov 03 13:39:14.545574 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.545544    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"
Nov 03 13:39:14.545794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.545585    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" event={"ID":"5baee21a-8b1f-4942-9593-5104f2981e18","Type":"ContainerDied","Data":"40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58"}
Nov 03 13:39:14.545794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.545637    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt" event={"ID":"5baee21a-8b1f-4942-9593-5104f2981e18","Type":"ContainerDied","Data":"c4442d68a51a74c56c492e9e509f5e17494c4c1870459dd6e23bba9225300ed6"}
Nov 03 13:39:14.545794 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.545659    4909 scope.go:122] "RemoveContainer" containerID="40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58"
Nov 03 13:39:14.555136 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.555119    4909 scope.go:122] "RemoveContainer" containerID="40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58"
Nov 03 13:39:14.555393 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:39:14.555375    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58\": container with ID starting with 40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58 not found: ID does not exist" containerID="40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58"
Nov 03 13:39:14.555443 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.555402    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58"} err="failed to get container status \"40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58\": rpc error: code = NotFound desc = could not find container \"40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58\": container with ID starting with 40f6c960c2dd20cff98b4e99fe47e620f8a5d5de8a65c2966360b56d63213a58 not found: ID does not exist"
Nov 03 13:39:14.697847 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.697818    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"]
Nov 03 13:39:14.743578 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:14.743549    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-tcqvt"]
Nov 03 13:39:15.121881 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:15.121850    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5baee21a-8b1f-4942-9593-5104f2981e18" path="/var/lib/kubelet/pods/5baee21a-8b1f-4942-9593-5104f2981e18/volumes"
Nov 03 13:39:22.817042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:22.817007    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:39:39.563262 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:39:39.563172    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:39:49.580054 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.580017    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"]
Nov 03 13:39:49.580458 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.580413    4909 cpu_manager.go:414] "containerMap: removing container" podUID="5baee21a-8b1f-4942-9593-5104f2981e18" containerName="maas-api"
Nov 03 13:39:49.580458 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.580424    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baee21a-8b1f-4942-9593-5104f2981e18" containerName="maas-api"
Nov 03 13:39:49.580536 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.580520    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baee21a-8b1f-4942-9593-5104f2981e18" containerName="maas-api"
Nov 03 13:39:49.583591 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.583566    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.655673 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.655640    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e051d-0663-4d30-b14f-538a1255518a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.655840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.655714    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.655892 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.655833    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.655972 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.655951    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dkl\" (UniqueName: \"kubernetes.io/projected/bd3e051d-0663-4d30-b14f-538a1255518a-kube-api-access-m9dkl\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.679819 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.679788    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"]
Nov 03 13:39:49.716323 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.716299    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a6706a-serving-cert\""
Nov 03 13:39:49.757145 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.757116    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e051d-0663-4d30-b14f-538a1255518a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.757279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.757162    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.757279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.757201    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.757279 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.757242    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dkl\" (UniqueName: \"kubernetes.io/projected/bd3e051d-0663-4d30-b14f-538a1255518a-kube-api-access-m9dkl\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.757446 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:39:49.757344    4909 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-serving-cert: secret "maas-api-e2e-worker-single-main-a6706a-serving-cert" not found
Nov 03 13:39:49.757446 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:39:49.757411    4909 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls podName:bd3e051d-0663-4d30-b14f-538a1255518a nodeName:}" failed. No retries permitted until 2026-11-03 13:39:50.257389408 +0000 UTC m=+6539.269856067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls") pod "maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" (UID: "bd3e051d-0663-4d30-b14f-538a1255518a") : secret "maas-api-e2e-worker-single-main-a6706a-serving-cert" not found
Nov 03 13:39:49.757872 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.757854    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e051d-0663-4d30-b14f-538a1255518a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.759739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.759721    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:49.819024 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:49.818987    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dkl\" (UniqueName: \"kubernetes.io/projected/bd3e051d-0663-4d30-b14f-538a1255518a-kube-api-access-m9dkl\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:50.262731 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:50.262682    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:50.265102 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:50.265079    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:50.496635 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:50.496589    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:50.875523 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:50.875496    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"]
Nov 03 13:39:50.876200 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:39:50.876174    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3e051d_0663_4d30_b14f_538a1255518a.slice/crio-2fd0853fd97eda555025206cf773b9af6d5986d27d972e9bbce2d0d4ca4386dd WatchSource:0}: Error finding container 2fd0853fd97eda555025206cf773b9af6d5986d27d972e9bbce2d0d4ca4386dd: Status 404 returned error can't find the container with id 2fd0853fd97eda555025206cf773b9af6d5986d27d972e9bbce2d0d4ca4386dd
Nov 03 13:39:51.718357 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:51.718317    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" event={"ID":"bd3e051d-0663-4d30-b14f-538a1255518a","Type":"ContainerStarted","Data":"2fd0853fd97eda555025206cf773b9af6d5986d27d972e9bbce2d0d4ca4386dd"}
Nov 03 13:39:52.724953 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:52.724913    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" event={"ID":"bd3e051d-0663-4d30-b14f-538a1255518a","Type":"ContainerStarted","Data":"9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c"}
Nov 03 13:39:52.725433 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:52.725053    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:52.844435 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:52.844379    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" podStartSLOduration=2.947597019 podStartE2EDuration="3.844363301s" totalImagesPullingTime="896.766282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:52.842834965 +0000 UTC m=+6541.855301630" watchObservedRunningTime="2026-11-03 13:39:52.844363301 +0000 UTC m=+6541.856829960"
Nov 03 13:39:58.742059 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:58.742027    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:39:58.745488 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:39:58.745461    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:40:09.564521 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:40:09.564484    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:40:18.569551 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.569511    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"]
Nov 03 13:40:18.570042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.569773    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" podUID="bd3e051d-0663-4d30-b14f-538a1255518a" containerName="maas-api" containerID="cri-o://9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c" gracePeriod=30
Nov 03 13:40:18.729919 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.729883    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" podUID="bd3e051d-0663-4d30-b14f-538a1255518a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.237:8443/health\": dial tcp 10.132.0.237:8443: connect: connection refused"
Nov 03 13:40:18.834393 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.834372    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:40:18.836482 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.836459    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-metrics-certs\") pod \"bd3e051d-0663-4d30-b14f-538a1255518a\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") "
Nov 03 13:40:18.836573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.836499    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3e051d-0663-4d30-b14f-538a1255518a-kube-api-access-m9dkl\" (UniqueName: \"kubernetes.io/projected/bd3e051d-0663-4d30-b14f-538a1255518a-kube-api-access-m9dkl\") pod \"bd3e051d-0663-4d30-b14f-538a1255518a\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") "
Nov 03 13:40:18.836573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.836537    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3e051d-0663-4d30-b14f-538a1255518a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e051d-0663-4d30-b14f-538a1255518a-openshift-service-ca\") pod \"bd3e051d-0663-4d30-b14f-538a1255518a\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") "
Nov 03 13:40:18.836741 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.836587    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls\") pod \"bd3e051d-0663-4d30-b14f-538a1255518a\" (UID: \"bd3e051d-0663-4d30-b14f-538a1255518a\") "
Nov 03 13:40:18.836980 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.836952    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3e051d-0663-4d30-b14f-538a1255518a-openshift-service-ca" pod "bd3e051d-0663-4d30-b14f-538a1255518a" (UID: "bd3e051d-0663-4d30-b14f-538a1255518a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:18.838739 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.838717    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls" pod "bd3e051d-0663-4d30-b14f-538a1255518a" (UID: "bd3e051d-0663-4d30-b14f-538a1255518a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:18.838847 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.838790    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3e051d-0663-4d30-b14f-538a1255518a-kube-api-access-m9dkl" pod "bd3e051d-0663-4d30-b14f-538a1255518a" (UID: "bd3e051d-0663-4d30-b14f-538a1255518a"). InnerVolumeSpecName "kube-api-access-m9dkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:18.838898 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.838874    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-metrics-certs" pod "bd3e051d-0663-4d30-b14f-538a1255518a" (UID: "bd3e051d-0663-4d30-b14f-538a1255518a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:18.840789 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.840766    4909 generic.go:356] "Generic (PLEG): container finished" podID="bd3e051d-0663-4d30-b14f-538a1255518a" containerID="9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c" exitCode=0
Nov 03 13:40:18.840891 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.840815    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"
Nov 03 13:40:18.840954 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.840812    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" event={"ID":"bd3e051d-0663-4d30-b14f-538a1255518a","Type":"ContainerDied","Data":"9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c"}
Nov 03 13:40:18.841019 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.840958    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj" event={"ID":"bd3e051d-0663-4d30-b14f-538a1255518a","Type":"ContainerDied","Data":"2fd0853fd97eda555025206cf773b9af6d5986d27d972e9bbce2d0d4ca4386dd"}
Nov 03 13:40:18.841019 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.840995    4909 scope.go:122] "RemoveContainer" containerID="9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c"
Nov 03 13:40:18.860458 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.860443    4909 scope.go:122] "RemoveContainer" containerID="9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c"
Nov 03 13:40:18.860745 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:40:18.860726    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c\": container with ID starting with 9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c not found: ID does not exist" containerID="9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c"
Nov 03 13:40:18.860844 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.860751    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c"} err="failed to get container status \"9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c\": rpc error: code = NotFound desc = could not find container \"9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c\": container with ID starting with 9d57f6efbba66c45b0b9f980565b4a47e49ae7d0d5c69496458a925044ab218c not found: ID does not exist"
Nov 03 13:40:18.937605 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.937583    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:18.937605 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.937604    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dkl\" (UniqueName: \"kubernetes.io/projected/bd3e051d-0663-4d30-b14f-538a1255518a-kube-api-access-m9dkl\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:18.937787 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.937615    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e051d-0663-4d30-b14f-538a1255518a-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:18.937787 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:18.937625    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3e051d-0663-4d30-b14f-538a1255518a-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:19.060007 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:19.059976    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"]
Nov 03 13:40:19.062940 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:19.062916    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a6706a-5b8586fbdf-m28cj"]
Nov 03 13:40:19.127629 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:19.127534    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3e051d-0663-4d30-b14f-538a1255518a" path="/var/lib/kubelet/pods/bd3e051d-0663-4d30-b14f-538a1255518a/volumes"
Nov 03 13:40:24.933246 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:24.933214    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:40:37.590480 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:37.590448    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:40:39.687725 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:40:39.685999    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:40:51.963267 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:51.963233    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:40:51.978593 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:40:51.978573    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:41:09.563800 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:41:09.563763    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:41:39.564211 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:41:39.564168    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:42:08.990896 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:08.990845    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v5l4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:42:08.991426 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:08.990929    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v5l4l" podUID="85e0502c-2cd5-4ee7-a0cf-6807f9c3f597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:42:09.562945 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:42:09.562909    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:42:10.529242 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:10.529208    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:42:22.939221 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:22.939187    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:42:32.188414 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:32.188334    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:42:39.563721 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:42:39.563661    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:42:40.200265 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:40.200230    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:42:45.126253 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.126221    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"]
Nov 03 13:42:45.126670 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.126657    4909 cpu_manager.go:414] "containerMap: removing container" podUID="bd3e051d-0663-4d30-b14f-538a1255518a" containerName="maas-api"
Nov 03 13:42:45.126737 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.126671    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3e051d-0663-4d30-b14f-538a1255518a" containerName="maas-api"
Nov 03 13:42:45.126847 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.126836    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3e051d-0663-4d30-b14f-538a1255518a" containerName="maas-api"
Nov 03 13:42:45.130278 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.130257    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.226526 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.226497    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"]
Nov 03 13:42:45.247790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.247764    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6fn\" (UniqueName: \"kubernetes.io/projected/521d1ded-bc9b-4ef5-a957-5863e53da561-kube-api-access-6v6fn\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.247923 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.247843    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d1ded-bc9b-4ef5-a957-5863e53da561-openshift-service-ca\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.247923 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.247891    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d1ded-bc9b-4ef5-a957-5863e53da561-metrics-certs\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.248003 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.247933    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d1ded-bc9b-4ef5-a957-5863e53da561-maas-api-tls\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.349442 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.349410    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6fn\" (UniqueName: \"kubernetes.io/projected/521d1ded-bc9b-4ef5-a957-5863e53da561-kube-api-access-6v6fn\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.349615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.349455    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d1ded-bc9b-4ef5-a957-5863e53da561-openshift-service-ca\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.349615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.349477    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d1ded-bc9b-4ef5-a957-5863e53da561-metrics-certs\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.349615 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.349513    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d1ded-bc9b-4ef5-a957-5863e53da561-maas-api-tls\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.350192 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.350167    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d1ded-bc9b-4ef5-a957-5863e53da561-openshift-service-ca\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.352073 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.352042    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d1ded-bc9b-4ef5-a957-5863e53da561-metrics-certs\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.352162 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.352071    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d1ded-bc9b-4ef5-a957-5863e53da561-maas-api-tls\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.401985 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.401928    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6fn\" (UniqueName: \"kubernetes.io/projected/521d1ded-bc9b-4ef5-a957-5863e53da561-kube-api-access-6v6fn\") pod \"maas-api-6c989b4c99-47shx\" (UID: \"521d1ded-bc9b-4ef5-a957-5863e53da561\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.443202 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.443179    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:45.804184 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:45.804149    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"]
Nov 03 13:42:45.807244 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:42:45.807215    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521d1ded_bc9b_4ef5_a957_5863e53da561.slice/crio-24d45bc53b525fbf46ed6e7992ec970874bf8b5b23ba7ea55d96de1316d58cea WatchSource:0}: Error finding container 24d45bc53b525fbf46ed6e7992ec970874bf8b5b23ba7ea55d96de1316d58cea: Status 404 returned error can't find the container with id 24d45bc53b525fbf46ed6e7992ec970874bf8b5b23ba7ea55d96de1316d58cea
Nov 03 13:42:46.675301 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:46.675266    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx" event={"ID":"521d1ded-bc9b-4ef5-a957-5863e53da561","Type":"ContainerStarted","Data":"24d45bc53b525fbf46ed6e7992ec970874bf8b5b23ba7ea55d96de1316d58cea"}
Nov 03 13:42:47.681870 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:47.681827    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx" event={"ID":"521d1ded-bc9b-4ef5-a957-5863e53da561","Type":"ContainerStarted","Data":"7a515d65f3bcd3e0e8fe693483e2f80efdd856e623be640c262c2277b1cf5fca"}
Nov 03 13:42:47.682291 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:47.681904    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:47.790573 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:47.790513    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx" podStartSLOduration=1.861912802 podStartE2EDuration="2.790497843s" totalImagesPullingTime="928.585041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:47.789288966 +0000 UTC m=+6716.801755628" watchObservedRunningTime="2026-11-03 13:42:47.790497843 +0000 UTC m=+6716.802964507"
Nov 03 13:42:53.692386 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:53.692356    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-47shx"
Nov 03 13:42:57.637760 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:42:57.637727    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:43:02.005748 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:02.005691    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"]
Nov 03 13:43:02.006174 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:02.005892    4909 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh" podUID="1adbd0bd-681f-4db9-9ddc-1563d890c150" containerName="maas-api" containerID="cri-o://b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90" gracePeriod=30
Nov 03 13:43:03.253914 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.253888    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 13:43:03.350753 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.350672    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adbd0bd-681f-4db9-9ddc-1563d890c150-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbd0bd-681f-4db9-9ddc-1563d890c150-openshift-service-ca\") pod \"1adbd0bd-681f-4db9-9ddc-1563d890c150\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") "
Nov 03 13:43:03.350887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.350765    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-maas-api-tls\") pod \"1adbd0bd-681f-4db9-9ddc-1563d890c150\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") "
Nov 03 13:43:03.350887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.350800    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-metrics-certs\") pod \"1adbd0bd-681f-4db9-9ddc-1563d890c150\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") "
Nov 03 13:43:03.350984 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.350901    4909 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adbd0bd-681f-4db9-9ddc-1563d890c150-kube-api-access-lvlsj\" (UniqueName: \"kubernetes.io/projected/1adbd0bd-681f-4db9-9ddc-1563d890c150-kube-api-access-lvlsj\") pod \"1adbd0bd-681f-4db9-9ddc-1563d890c150\" (UID: \"1adbd0bd-681f-4db9-9ddc-1563d890c150\") "
Nov 03 13:43:03.351073 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.351030    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adbd0bd-681f-4db9-9ddc-1563d890c150-openshift-service-ca" pod "1adbd0bd-681f-4db9-9ddc-1563d890c150" (UID: "1adbd0bd-681f-4db9-9ddc-1563d890c150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:03.351359 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.351341    4909 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbd0bd-681f-4db9-9ddc-1563d890c150-openshift-service-ca\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:03.353002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.352977    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-metrics-certs" pod "1adbd0bd-681f-4db9-9ddc-1563d890c150" (UID: "1adbd0bd-681f-4db9-9ddc-1563d890c150"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:03.353002 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.352988    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-maas-api-tls" pod "1adbd0bd-681f-4db9-9ddc-1563d890c150" (UID: "1adbd0bd-681f-4db9-9ddc-1563d890c150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:03.353138 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.353024    4909 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adbd0bd-681f-4db9-9ddc-1563d890c150-kube-api-access-lvlsj" pod "1adbd0bd-681f-4db9-9ddc-1563d890c150" (UID: "1adbd0bd-681f-4db9-9ddc-1563d890c150"). InnerVolumeSpecName "kube-api-access-lvlsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:03.452187 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.452159    4909 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-maas-api-tls\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:03.452187 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.452184    4909 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adbd0bd-681f-4db9-9ddc-1563d890c150-metrics-certs\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:03.452363 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.452194    4909 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvlsj\" (UniqueName: \"kubernetes.io/projected/1adbd0bd-681f-4db9-9ddc-1563d890c150-kube-api-access-lvlsj\") on node \"ip-10-0-131-136.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:03.780881 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.780847    4909 generic.go:356] "Generic (PLEG): container finished" podID="1adbd0bd-681f-4db9-9ddc-1563d890c150" containerID="b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90" exitCode=0
Nov 03 13:43:03.781079 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.780901    4909 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"
Nov 03 13:43:03.781079 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.780913    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh" event={"ID":"1adbd0bd-681f-4db9-9ddc-1563d890c150","Type":"ContainerDied","Data":"b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90"}
Nov 03 13:43:03.781079 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.780955    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh" event={"ID":"1adbd0bd-681f-4db9-9ddc-1563d890c150","Type":"ContainerDied","Data":"3ed5db4c0c3017f77856aa89a98cf8ff24ece7cfaa8ceecf7d5b93e1b9e38e1a"}
Nov 03 13:43:03.781079 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.780975    4909 scope.go:122] "RemoveContainer" containerID="b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90"
Nov 03 13:43:03.790681 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.790663    4909 scope.go:122] "RemoveContainer" containerID="b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90"
Nov 03 13:43:03.790993 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:43:03.790971    4909 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90\": container with ID starting with b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90 not found: ID does not exist" containerID="b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90"
Nov 03 13:43:03.791092 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.791000    4909 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90"} err="failed to get container status \"b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90\": rpc error: code = NotFound desc = could not find container \"b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90\": container with ID starting with b659c8c94d9d17675b089c9422cdbcea85181e9cefd253e555ca5225a919fd90 not found: ID does not exist"
Nov 03 13:43:03.932498 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.932471    4909 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"]
Nov 03 13:43:03.937028 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:03.937007    4909 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-phtbh"]
Nov 03 13:43:05.126137 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:05.126097    4909 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adbd0bd-681f-4db9-9ddc-1563d890c150" path="/var/lib/kubelet/pods/1adbd0bd-681f-4db9-9ddc-1563d890c150/volumes"
Nov 03 13:43:09.564584 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:43:09.564545    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:43:34.492037 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:34.492002    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:43:39.562521 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:43:39.562484    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:43:46.226616 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:43:46.226581    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:44:07.713215 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:44:07.713117    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:44:09.565865 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:44:09.565830    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:44:19.646178 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:44:19.646120    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:44:39.563022 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:44:39.562987    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:44:50.293006 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:44:50.292967    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:45:01.017125 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:01.017090    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:45:03.126723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:03.125786    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:45:03.126723 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:03.125852    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:45:03.220840 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:03.220803    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:45:03.221143 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:03.221109    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:45:09.693722 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:45:09.693497    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:45:14.826459 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:14.826423    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:45:25.723547 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:25.723511    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:45:39.564082 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:45:39.564001    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:45:49.100455 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:49.100420    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:45:52.128814 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:52.124392    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:45:52.128814 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:52.126358    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rmmzr_70ab9d5e-f98a-4b5a-b72b-175e9182f488/console-operator/2.log"
Nov 03 13:45:59.825080 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:45:59.825045    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:46:09.225379 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:46:09.225341    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:46:09.565664 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:46:09.565585    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:46:29.427994 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:46:29.427958    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:46:39.564383 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:46:39.564337    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:46:49.813597 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:46:49.813561    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:47:09.563462 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:47:09.563416    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:47:14.233587 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:47:14.233553    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bwfnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:47:14.234235 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:47:14.234201    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bwfnv" podUID="925da08a-493d-409c-8476-7721001ae76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:47:39.562953 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:47:39.562916    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:48:09.563785 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:48:09.563745    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:48:18.123818 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:48:18.123779    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f2bh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:48:18.124554 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:48:18.124518    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f2bh6" podUID="858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:48:18.236880 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:48:18.236832    4909 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vr8qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:48:18.237053 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:48:18.236908    4909 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vr8qh" podUID="98367206-ed8d-4a86-b2e0-5850d8e470e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:48:39.562520 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:48:39.562440    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:48:47.320860 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:48:47.320822    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xj776"]
Nov 03 13:49:09.562648 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:49:09.562613    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:49:16.665030 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:16.664991    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-csc6h_e135ffb3-aaf9-493b-b0ba-87d09909709e/server/0.log"
Nov 03 13:49:16.902438 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:16.902401    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cbwwg_d66fb109-6507-43fe-8916-82bcc32b4ecc/manager/0.log"
Nov 03 13:49:18.173451 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:18.173422    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c989b4c99-47shx_521d1ded-bc9b-4ef5-a957-5863e53da561/maas-api/0.log"
Nov 03 13:49:25.623934 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:25.623890    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xj776_ba0dd0de-32a0-4a53-a8ab-3c8804f7af66/limitador/0.log"
Nov 03 13:49:27.069230 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:27.069190    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-klwdt_b3f9e19a-8336-4fe1-8ac3-721dcf430675/discovery/0.log"
Nov 03 13:49:27.280056 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:27.280018    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758d88fbcf-b96gm_93aba9c3-9358-4bc6-bd3b-69d1a48bb631/kube-auth-proxy/0.log"
Nov 03 13:49:27.713045 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:27.713013    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-bwfnv_925da08a-493d-409c-8476-7721001ae76f/istio-proxy/0.log"
Nov 03 13:49:28.176081 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:28.175972    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-f2bh6_858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d/istio-proxy/0.log"
Nov 03 13:49:29.152219 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:29.152191    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-v5l4l_85e0502c-2cd5-4ee7-a0cf-6807f9c3f597/istio-proxy/0.log"
Nov 03 13:49:29.383258 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:29.383229    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vr8qh_98367206-ed8d-4a86-b2e0-5850d8e470e1/istio-proxy/0.log"
Nov 03 13:49:30.554213 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:30.554167    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f7b9fc877-jqd6f_6cbdbdac-bc9c-4945-8375-4aaac24d79e0/payload-processing/0.log"
Nov 03 13:49:30.771026 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:30.770991    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b5bc54f87-gkx8h_71b27f5e-a0e1-4ba7-a4af-8282a4639d41/router/0.log"
Nov 03 13:49:32.072073 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:32.072026    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-l4w58_a893c515-62e2-4508-a564-127448f1d80f/main/0.log"
Nov 03 13:49:32.335282 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:32.335205    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9xcdv_7892de29-7641-4f9a-8e86-787c3e3c40bb/main/0.log"
Nov 03 13:49:32.774182 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:32.774154    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pdsk_6e3dcfeb-8478-4ad9-b1cf-c2690a1a6582/main/0.log"
Nov 03 13:49:32.986769 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:32.986738    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-f98p8_1efe5333-ea36-4b1b-8369-e71b32c08e7c/main/0.log"
Nov 03 13:49:39.562753 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:49:39.562713    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:49:54.281868 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:54.281839    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lsvz7_098926dd-2788-41f4-a964-5f7448c8c41f/global-pull-secret-syncer/0.log"
Nov 03 13:49:54.863597 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:54.863568    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-n7zd8_7b7eef80-4dd6-4087-b74c-66adfb323841/connection-checker/0.log"
Nov 03 13:49:55.287457 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:55.287430    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nct5z_9e19744a-acf7-4e34-957f-c01efa20f0b2/connection-checker/0.log"
Nov 03 13:49:55.696103 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:55.696078    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xv5pl_8e156d6b-a0b5-460d-8946-7f7329753e54/connection-checker/0.log"
Nov 03 13:49:56.556675 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:56.556646    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-df7t5_77d106d7-59a7-459c-b012-aaa087cb50d9/konnectivity-agent/0.log"
Nov 03 13:49:57.466259 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:49:57.466201    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-136.us-west-2.compute.internal_f2905ce8bf73baad4434061659facba8/haproxy/0.log"
Nov 03 13:50:04.710410 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.710328    4909 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"]
Nov 03 13:50:04.710921 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.710742    4909 cpu_manager.go:414] "containerMap: removing container" podUID="1adbd0bd-681f-4db9-9ddc-1563d890c150" containerName="maas-api"
Nov 03 13:50:04.710921 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.710754    4909 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adbd0bd-681f-4db9-9ddc-1563d890c150" containerName="maas-api"
Nov 03 13:50:04.710921 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.710835    4909 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adbd0bd-681f-4db9-9ddc-1563d890c150" containerName="maas-api"
Nov 03 13:50:04.714042 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.714021    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.762359 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.762336    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fst8\"/\"kube-root-ca.crt\""
Nov 03 13:50:04.808317 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.808291    4909 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7fst8\"/\"default-dockercfg-mjtw8\""
Nov 03 13:50:04.810075 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.810044    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"]
Nov 03 13:50:04.838425 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.838401    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-sys\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.838566 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.838441    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pv6b\" (UniqueName: \"kubernetes.io/projected/3aea3541-3e62-46ef-b014-17e83bb9d820-kube-api-access-9pv6b\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.838638 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.838573    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-lib-modules\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.838728 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.838682    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-proc\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.838812 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.838751    4909 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-podres\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.863338 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.863316    4909 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fst8\"/\"openshift-service-ca.crt\""
Nov 03 13:50:04.939563 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939540    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-lib-modules\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939673 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939594    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-proc\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939673 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939640    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-podres\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939671    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-sys\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939683    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-lib-modules\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939742    4909 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pv6b\" (UniqueName: \"kubernetes.io/projected/3aea3541-3e62-46ef-b014-17e83bb9d820-kube-api-access-9pv6b\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939753    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-proc\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939756    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-podres\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.939790 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.939772    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aea3541-3e62-46ef-b014-17e83bb9d820-sys\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:04.993148 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:04.993093    4909 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pv6b\" (UniqueName: \"kubernetes.io/projected/3aea3541-3e62-46ef-b014-17e83bb9d820-kube-api-access-9pv6b\") pod \"perf-node-gather-daemonset-4z726\" (UID: \"3aea3541-3e62-46ef-b014-17e83bb9d820\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:05.026481 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:05.026457    4909 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:05.391424 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:05.391398    4909 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"]
Nov 03 13:50:05.393859 ip-10-0-131-136 kubenswrapper[4909]: W1103 13:50:05.393830    4909 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3aea3541_3e62_46ef_b014_17e83bb9d820.slice/crio-f8e67cfa0ed15f337fc1d0a6709832208e93fb1ea47645bd937a89a226b228a2 WatchSource:0}: Error finding container f8e67cfa0ed15f337fc1d0a6709832208e93fb1ea47645bd937a89a226b228a2: Status 404 returned error can't find the container with id f8e67cfa0ed15f337fc1d0a6709832208e93fb1ea47645bd937a89a226b228a2
Nov 03 13:50:05.867541 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:05.867504    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-csc6h_e135ffb3-aaf9-493b-b0ba-87d09909709e/server/0.log"
Nov 03 13:50:06.109415 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:06.109382    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xj776_ba0dd0de-32a0-4a53-a8ab-3c8804f7af66/limitador/0.log"
Nov 03 13:50:06.285783 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:06.285748    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726" event={"ID":"3aea3541-3e62-46ef-b014-17e83bb9d820","Type":"ContainerStarted","Data":"a6763741a07f3a6518d51727bb2a6efbffaba643ac7c9300dc6cf99201e912b5"}
Nov 03 13:50:06.285974 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:06.285953    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:06.286037 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:06.285990    4909 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726" event={"ID":"3aea3541-3e62-46ef-b014-17e83bb9d820","Type":"ContainerStarted","Data":"f8e67cfa0ed15f337fc1d0a6709832208e93fb1ea47645bd937a89a226b228a2"}
Nov 03 13:50:06.394906 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:06.394855    4909 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726" podStartSLOduration=2.394841766 podStartE2EDuration="2.394841766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:06.393279368 +0000 UTC m=+7155.405746031" watchObservedRunningTime="2026-11-03 13:50:06.394841766 +0000 UTC m=+7155.407308464"
Nov 03 13:50:07.008567 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:07.008536    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cbwwg_d66fb109-6507-43fe-8916-82bcc32b4ecc/manager/0.log"
Nov 03 13:50:09.563149 ip-10-0-131-136 kubenswrapper[4909]: E1103 13:50:09.563109    4909 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bq6v7" containerName="extractor"
Nov 03 13:50:11.687302 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:11.687270    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-csc6h_e135ffb3-aaf9-493b-b0ba-87d09909709e/server/0.log"
Nov 03 13:50:12.156128 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:12.156092    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-xtwgh_cb95efe0-e733-48c0-9ff5-a92a0f1ae4e8/cluster-monitoring-operator/0.log"
Nov 03 13:50:12.301675 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:12.301645    4909 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-4z726"
Nov 03 13:50:12.866265 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:12.866230    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cbwwg_d66fb109-6507-43fe-8916-82bcc32b4ecc/manager/0.log"
Nov 03 13:50:14.578001 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:14.577971    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dzlns_d0834d68-fe3a-4284-868d-04034a3704b3/node-exporter/0.log"
Nov 03 13:50:14.985289 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:14.985261    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dzlns_d0834d68-fe3a-4284-868d-04034a3704b3/kube-rbac-proxy/0.log"
Nov 03 13:50:15.389839 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:15.389769    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dzlns_d0834d68-fe3a-4284-868d-04034a3704b3/init-textfile/0.log"
Nov 03 13:50:17.759425 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:17.759398    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-csc6h_e135ffb3-aaf9-493b-b0ba-87d09909709e/server/0.log"
Nov 03 13:50:18.936427 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:18.936397    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cbwwg_d66fb109-6507-43fe-8916-82bcc32b4ecc/manager/0.log"
Nov 03 13:50:24.824515 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:24.824482    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-klwdt_b3f9e19a-8336-4fe1-8ac3-721dcf430675/discovery/0.log"
Nov 03 13:50:24.934122 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:24.934094    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758d88fbcf-b96gm_93aba9c3-9358-4bc6-bd3b-69d1a48bb631/kube-auth-proxy/0.log"
Nov 03 13:50:25.371421 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:25.371393    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-bwfnv_925da08a-493d-409c-8476-7721001ae76f/istio-proxy/0.log"
Nov 03 13:50:25.804887 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:25.804856    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-f2bh6_858aa31d-4b48-4cc0-a6d7-e4e9ba1ddd5d/istio-proxy/0.log"
Nov 03 13:50:26.680543 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:26.680516    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-v5l4l_85e0502c-2cd5-4ee7-a0cf-6807f9c3f597/istio-proxy/0.log"
Nov 03 13:50:26.894258 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:26.894230    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vr8qh_98367206-ed8d-4a86-b2e0-5850d8e470e1/istio-proxy/0.log"
Nov 03 13:50:27.979217 ip-10-0-131-136 kubenswrapper[4909]: I1103 13:50:27.979174    4909 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f7b9fc877-jqd6f_6cbdbdac-bc9c-4945-8375-4aaac24d79e0/payload-processing/0.log"